Virtual guidance for ankle surgery procedures

ABSTRACT

An example method includes registering, via a visualization device, a virtual model of a portion of an anatomy of an ankle of a patient to a corresponding portion of the anatomy of the ankle viewable via the visualization device, the virtual model obtained from a virtual surgical plan for an ankle arthroplasty procedure to attach a prosthetic to the anatomy. The example method also comprises displaying, via the visualization device and overlaid on the portion of the anatomy, a virtual guide that guides at least one of preparation of the anatomy for attachment of the prosthetic or attachment of the prosthetic to the anatomy.

This patent application is a continuation of International ApplicationNo. PCT/US2019/036976, filed Jun. 13, 2019, which claims the benefit ofU.S. Provisional Patent Application No. 62/687,014, filed Jun. 19, 2018,U.S. Provisional Patent Application No. 62/739,406, filed Oct. 1, 2018,U.S. Provisional Patent Application No. 62/778,774, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,789, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,764, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,797, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,778, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,788, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,760, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,772, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,796, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,782, filed Dec. 12, 2018,U.S. Provisional Patent Application No. 62/778,791, filed Dec. 12, 2018,U.S. Provisional Patent Application 62/804,383, filed Feb. 12, 2019,U.S. Provisional Patent Application 62/804,392, filed Feb. 12, 2019, andU.S. Provisional Patent Application 62/804,402, filed Feb. 12, 2019.

BACKGROUND

Surgical joint repair procedures involve repair and/or replacement of adamaged or diseased joint. Many times, a surgical joint repairprocedure, such as joint arthroplasty as an example, involves replacingthe damaged joint with a prosthetic that is implanted into the patient'sbone. Proper selection of a prosthetic that is appropriately sized andshaped and proper positioning of that prosthetic to ensure an optimalsurgical outcome can be challenging. To assist with positioning, thesurgical procedure often involves the use of surgical instruments tocontrol the shaping of the surface of the damaged bone and cutting ordrilling of bone to accept the prosthetic.

Today, virtual visualization tools are available to surgeons that usethree-dimensional modeling of bone shapes to facilitate preoperativeplanning for joint repairs and replacements. These tools can assistsurgeons with the design and/or selection of surgical guides andimplants that closely match the patient's anatomy and can improvesurgical outcomes by customizing a surgical plan for each patient.

SUMMARY

This disclosure describes a variety of techniques for providingpreoperative planning, medical implant design and manufacture,intraoperative guidance, postoperative analysis, and/or training andeducation for surgical joint repair procedures. The techniques may beused independently or in various combinations to support particularphases or settings for surgical joint repair procedures or provide amulti-faceted ecosystem to support surgical joint repair procedures. Invarious examples, the disclosure describes techniques for preoperativesurgical planning, intra-operative surgical planning, intra-operativesurgical guidance, intra-operative surgical tracking and post-operativeanalysis using mixed reality (MR)-based visualization. In some examples,the disclosure also describes surgical items and/or methods forperforming surgical joint repair procedures. In some examples, thisdisclosure also describes techniques and visualization devicesconfigured to provide education about an orthopedic surgical procedureusing mixed reality.

The details of various examples of the disclosure are set forth in theaccompanying drawings and the description below. Various features,objects, and advantages will be apparent from the description, drawings,and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an orthopedic surgical system according toan example of this disclosure.

FIG. 2 is a block diagram of an orthopedic surgical system that includesa mixed reality (MR) system, according to an example of this disclosure.

FIG. 3 is a flowchart illustrating example phases of a surgicallifecycle.

FIG. 4 is a flowchart illustrating preoperative, intraoperative andpostoperative workflows in support of an orthopedic surgical procedure.

FIG. 5 is a schematic representation of a visualization device for usein a mixed reality (MR) system, according to an example of thisdisclosure.

FIG. 6 is a block diagram illustrating example components of avisualization device for use in a mixed reality (MR) system, accordingto an example of this disclosure.

FIG. 7 is a conceptual diagram illustrating an example setting in whicha set of users use mixed reality (MR) systems of an orthopedic surgicalsystem during a preoperative phase.

FIG. 8 is a flowchart illustrating example steps in the preoperativephase of the surgical lifecycle.

FIG. 9 illustrates an example welcome page for selecting a surgicalcase, according to an example of this disclosure.

FIG. 10 illustrates an example of a page of a user interface of a mixedreality (MR) system, according to an example of this disclosure.

FIG. 11A is an example of information that can be displayed on a page ofthe user interface of FIG. 10.

FIG. 11B is a flowchart illustrating an example operation to assist insurgical parameter selection, in accordance with a technique of thisdisclosure.

FIG. 12 illustrates effects of de-selection of icons in the navigationbar of FIG. 10.

FIG. 13 is an example of an Augment Surgery mode widget that isdisplayed on various pages of the user interface of FIG. 10, accordingto an example of this disclosure.

FIG. 14 is a conceptual diagram illustrating an example of informationdisplayed in a workflow page of the user interface of FIG. 10.

FIG. 15A is an example of a humerus cut page of the user interface ofFIG. 10, according to an example of this disclosure.

FIGS. 15B-15D are examples of hiding virtual objects in the humerus cutpage of FIG. 15A, according to an example of this disclosure.

FIG. 16 is an example of an install guide page of the user interface ofFIG. 10, according to an example of this disclosure.

FIG. 17 is an example of an install implant page of the user interfaceof FIG. 10, according to an example of this disclosure.

FIG. 18 is a conceptual diagram illustrating an example setting in whicha set of users use MR systems of an orthopedic surgical system during anintraoperative phase.

FIG. 19 is a flowchart illustrating example stages of a shoulder jointrepair surgery.

FIGS. 20A and 20B illustrate example techniques for registering a3-dimensional virtual bone model with an observed real bone structure ofa patient during joint repair surgery.

FIG. 21 is a conceptual diagram illustrating steps of an exampleregistration process for a shoulder arthroplasty procedure.

FIG. 22 is a conceptual diagram illustrating additional steps of theexample registration process of the shoulder arthroplasty procedure ofFIG. 21.

FIG. 23 and FIG. 24 are conceptual diagrams further illustrating anexample registration process for a shoulder arthroplasty procedure.

FIG. 25 is a conceptual diagram illustrating an example registrationprocedure using a virtual marker.

FIG. 26 is a conceptual diagram illustrating additional steps of theexample registration procedure of FIG. 25 using a virtual marker.

FIG. 27 illustrates an image perceptible to a user when in an augmentsurgery mode of a mixed reality (MR) system, according to an example ofthis disclosure.

FIG. 28 illustrates an example of virtual images that a surgeon can seeof implant components in an augment surgery mode of a mixed reality (MR)system.

FIG. 29 illustrates an example of virtual images that a surgeon can seeof implant components in an augment surgery mode of a mixed reality (MR)system.

FIGS. 30A-30E illustrate examples of physical markers that can beemployed in the mixed reality (MR) system of FIG. 1, according to anexample of this disclosure.

FIG. 31 is an example of a process flow for tracking in the augmentsurgery mode of the mixed reality (MR) system, according to an exampleof this disclosure.

FIGS. 32A-32C illustrate steps a surgeon may perform to resect a humeralhead of a humerus in a shoulder arthroplasty procedure.

FIG. 33 illustrates a mechanical guide for resection of the humeral headin a shoulder arthroplasty procedure.

FIGS. 34 and 35 are conceptual diagrams illustrating an MR systemproviding virtual guidance for installation of a mechanical guide in ahumeral head in a shoulder arthroplasty procedure, in accordance withone or more techniques of this disclosure.

FIGS. 36A-36D illustrate examples of virtual markers that an MR systemmay display.

FIGS. 37 and 38 are conceptual diagrams illustrating an MR systemproviding virtual guidance for reaming of a graft in a humeral head in ashoulder arthroplasty procedure, in accordance with one or moretechniques of this disclosure.

FIGS. 39 and 40 are conceptual diagrams illustrating an MR systemproviding virtual guidance for drilling a graft in a humeral head in ashoulder arthroplasty procedure, in accordance with one or moretechniques of this disclosure.

FIG. 41 is a conceptual diagram illustrating an MR system providingvirtual guidance for cutting of a graft in a humeral head in a shoulderarthroplasty procedure, in accordance with one or more techniques ofthis disclosure.

FIGS. 42A-42C are conceptual diagrams illustrating an MR systemproviding virtual guidance for resection of a humeral head in a shoulderarthroplasty procedure, in accordance with one or more techniques ofthis disclosure.

FIG. 43 is a conceptual diagram illustrating a physical guide forhumeral head resection that is positioned using virtual guidance in ashoulder arthroplasty procedure, in accordance with one or moretechniques of this disclosure.

FIGS. 44 and 45 are conceptual diagrams illustrating an MR systemproviding virtual guidance for creating a pilot hole in a humerus in ashoulder arthroplasty procedure, in accordance with one or moretechniques of this disclosure.

FIG. 46 is a conceptual diagram illustrating an MR system providingvirtual guidance for sounding a humerus in a shoulder arthroplastyprocedure, in accordance with one or more techniques of this disclosure.

FIG. 47 is a conceptual diagram illustrating an MR system providingvirtual guidance for punching a humerus in a shoulder arthroplastyprocedure, in accordance with one or more techniques of this disclosure.

FIG. 48 is a conceptual diagram illustrating an MR system providingvirtual guidance for compacting a humerus in a shoulder arthroplastyprocedure, in accordance with one or more techniques of this disclosure.

FIG. 49 is a conceptual diagram illustrating an MR system providingvirtual guidance for preparing a surface of a humerus in a shoulderarthroplasty procedure, in accordance with one or more techniques ofthis disclosure.

FIG. 50 is a conceptual diagram illustrating an MR system providingvirtual guidance for attaching an implant to a humerus in a shoulderarthroplasty procedure, in accordance with one or more techniques ofthis disclosure.

FIG. 51 is a conceptual diagram illustrating an MR system providingvirtual guidance to the user for installation of a guide in a glenoid ofa scapula in a shoulder arthroplasty procedure, in accordance with oneor more techniques of this disclosure.

FIG. 52 is a conceptual diagram illustrating an example guide asinstalled in a glenoid in a shoulder arthroplasty procedure.

FIG. 53 is a conceptual diagram illustrating an MR system providingvirtual guidance for reaming a glenoid in a shoulder arthroplastyprocedure, in accordance with one or more techniques of this disclosure.

FIGS. 54 and 55 are conceptual diagrams illustrating an MR systemproviding virtual guidance for creating a central hole in a glenoid(e.g., post-reaming) in a shoulder arthroplasty procedure, in accordancewith one or more techniques of this disclosure.

FIG. 56 is a conceptual diagram illustrating a glenoid prosthesis withkeel type anchorage.

FIGS. 57-59 are conceptual diagrams illustrating an MR system providingvirtual guidance for creating keel type anchorage positions in a glenoidin a shoulder arthroplasty procedure, in accordance with one or moretechniques of this disclosure.

FIG. 60 is a conceptual diagram illustrating a glenoid prosthesis withpegged type anchorage.

FIGS. 61 and 62 are conceptual diagrams illustrating an MR systemproviding virtual guidance for creating pegged type anchorage positionsin a glenoid in a shoulder arthroplasty procedure, in accordance withone or more techniques of this disclosure.

FIG. 63 is a conceptual diagram illustrating an MR system providingvirtual guidance for attaching an implant to a glenoid in a shoulderarthroplasty procedure, in accordance with one or more techniques ofthis disclosure.

FIGS. 64 and 65 illustrate screws and a central stem that may be used toattach a prothesis to a glenoid in a shoulder arthroplasty procedure.

FIG. 66 is a conceptual diagram of virtual guidance that may be providedby an MR system in a shoulder arthroplasty procedure, in accordance withone or more techniques of this disclosure.

FIG. 67 is a conceptual diagram of an example view that may be providedby an MR system in a shoulder arthroplasty procedure and that provides asecondary view window, in accordance with one or more techniques of thisdisclosure.

FIG. 68 is a conceptual diagram in which an MR system displays graphicaldepth guidance that includes an illustration that depicts a location ofa drill relative to a scapula in a shoulder arthroplasty procedure.

FIG. 69 is a conceptual diagram illustrating an example virtual modelwith a missing region surrounding an area of interest.

FIG. 70 is a conceptual diagram illustrating an example virtual modelwith a missing region surrounding an area of interest.

FIG. 71 is a conceptual diagram illustrating an example virtual modelwith a missing region surrounding an area of interest.

FIG. 72 is a flow diagram illustrating example techniques for closedloop tool control in surgical procedures, in accordance with one or moretechniques of this disclosure.

FIG. 73 is a conceptual side view of a portion of a medical device anddepth cameras consistent with an example of this disclosure.

FIG. 74 is a conceptual side view of a portion of a medical deviceconsistent with this disclosure.

FIG. 75 is a conceptual perspective view of a mixed reality (MR) systemand example reaming tool that includes a depth aid element consistentwith an example of this disclosure.

FIG. 76 is a conceptual perspective view of a mixed reality system thatmakes use of an example reaming tool that includes a depth aid elementconsistent with an example of this disclosure.

FIG. 77 is a flow diagram illustrating an example process consistentwith an example of this disclosure.

FIG. 78 is a side view of anther depth tracking example, which mayeliminate the need for a specially-designed depth aid element.

FIG. 79 is another side view showing an example of a depth aid elementattached at a fixed location relative to a reamer.

FIGS. 80, 81, and 82 are additional illustrations showing one exampleregistration process registering a location of a depth aid element.

FIG. 83 is a block diagram illustrating a system comprising a set ofsurgical items and a processing device in accordance with an example ofthis disclosure.

FIG. 84 is a conceptual block diagram illustrating a medical devicesystem comprising a set of surgical items and a processing device in theform of an MR visualization device in accordance with an example of thisdisclosure.

FIG. 85 is another conceptual block diagram illustrating a medicaldevice system comprising a set of surgical items and a processing devicein the form of an MR visualization device in accordance with an exampleof this disclosure.

FIG. 86 is a view of an example sounding procedure in a shoulderarthroplasty procedure.

FIG. 87 is a view of an example punching procedure in a shoulderarthroplasty procedure.

FIG. 88 is a view of an example compacting or rasping procedure in ashoulder arthroplasty procedure.

FIG. 89 is a view of an example surface planing procedure in a shoulderarthroplasty procedure.

FIG. 90 is a view of an example protect procedure in a shoulderarthroplasty procedure.

FIG. 91 and FIG. 92 are views of an example trial procedure in ashoulder arthroplasty procedure.

FIG. 93 and FIG. 94 are views of an example implant procedure in ashoulder arthroplasty procedure.

FIG. 95 is a conceptual side view of a sounding procedure beingperformed on a human humeral bone in which a sounder is inserted into apatient's humeral bone in a shoulder arthroplasty procedure.

FIG. 96 is an illustration of a set of medical sounders of differentsizes for use in the surgical procedure shown in FIG. 95.

FIG. 97 is an illustration of a set of tools, which include a set ofprogressively larger sounders.

FIG. 98 is a conceptual side view of a compacting or rasping procedurebeing performed on a human humeral bone.

FIG. 99 is an illustration of a set of tools, which include a set ofprogressively larger compacting tools.

FIG. 100 is a conceptual side view of a surface planing procedure beingperformed on a human humeral bone.

FIG. 101 is an illustration of a set of tools, which include a set ofprogressively larger surface planing tools.

FIG. 102 is a flow diagram illustrating a technique of identifying toolsin a surgical procedure.

FIG. 103 is a flow diagram illustrating a method for identifying toolsin a surgical procedure and changing the identification of such toolsbased on the stage of the surgical procedure.

FIG. 104 is a more detailed block diagram of a surgical item, which maycorrespond to one or more of the surgical items described in thisdisclosure.

FIG. 105 is an illustration of a set of surgical items similar to thoseillustrated in FIG. 97 with a virtual element presented to identify aparticular surgical item from the set.

FIG. 106 is another illustration of a set of surgical items similar tothose illustrated in FIG. 97 with a virtual element presented toidentify a particular surgical item from the set.

FIG. 107 is an illustration of a set of surgical items similar to thoseillustrated in FIG. 99 with virtual elements presented to identify itemsneeded in a surgical procedure, to identify items that require assemblyand to identify a side to be used on a two-sided item.

FIG. 108 is an illustration of a set of surgical items similar to thoseillustrated in FIG. 101 with virtual elements presented to identify aset of surgical items and to distinguish a given surgical item from theother identified surgical items.

FIG. 109 is block diagram illustrating an example system for generatingan extended reality (XR) visualization of a range of motion of anappendage of a patient, in accordance with a technique of thisdisclosure.

FIG. 110 is a conceptual diagram illustrating example motions of apatient's right arm that occur in the patient's shoulder.

FIG. 111 is a conceptual diagram illustrating an example extendedreality visualization of a range of motion, in accordance with atechnique of this disclosure.

FIG. 112A is a flowchart illustrating an example operation of a systemfor range of motion analysis and visualization, in accordance with atechnique of this disclosure.

FIG. 112B is a flowchart illustrating an example operation of a systemin accordance with a technique of this disclosure.

FIG. 113 is a conceptual diagram illustrating an example setting inwhich a set of users use MR systems for educational purposes.

FIG. 114 is a block diagram of a system that includes multiple MRdevices that communicate with one another.

FIG. 115 is a block diagram illustrating a distributed MR system thatincludes one or more users at a local environment that are incommunication with one or more users in a remote environment.

FIG. 116 is another block diagram illustrating an MR system thatincludes one or more users at a local environment that are incommunication with one or more users in a remote environment.

FIG. 117 is a block diagram illustrating an example system that mayassist a user, such as a surgeon, nurse, or other medical technicians,through steps in the workflow steps of orthopedic surgeries, inaccordance with a technique of this disclosure.

FIG. 118 is a flowchart illustrating an example operation to assistusers, such as surgeons, nurses, or other medical technicians, throughsteps in the workflows of orthopedic surgeries, in accordance with atechnique of this disclosure.

FIG. 119 is a conceptual diagram illustrating an example XRvisualization that includes a set of one or more virtual checklistitems, as viewed by a user, such as an orthopedic surgeon, nurse, orother medical technician, while performing an orthopedic surgery on ashoulder of a patient.

FIG. 120 is a conceptual diagram illustrating an example system in whicha first surgical plan is modified during an intraoperative phase togenerate a second surgical plan, in accordance with a technique of thisdisclosure.

FIG. 121 is a flowchart of an example operation in which a firstsurgical plan is modified during an intraoperative phase to generate asecond surgical plan, in accordance with a technique of this disclosure.

FIG. 122 is a block diagram illustrating an example computing systemthat implements a DNN usable for determining a classification of ashoulder condition of the patient, in accordance with a technique ofthis disclosure.

FIG. 123 illustrates an example deep neural network (DNN) that may beimplemented by a computing system with the system of FIG. 122.

FIG. 124 is a flowchart illustrating an example operation of a computingsystem in using a DNN to determine a classification of a shouldercondition, in accordance with a technique of this disclosure.

FIG. 125 is a block diagram illustrating example functional componentsof a computing system for using a DNN to determine a recommended surgeryfor a shoulder condition, in accordance with a technique of thisdisclosure.

FIG. 126 is a flowchart illustrating an example operation of a computingsystem that uses a DNN to determine a recommended type of shouldersurgery for a patient, in accordance with a technique of thisdisclosure.

FIG. 127 is a conceptual block diagram of an educational systemcomprising an MR teacher device and an MR student device for orthopedicsurgical education.

FIG. 128 is a conceptual block diagram of an educational systemcomprising an MR teacher device and a plurality of MR student devicesfor orthopedic surgical education.

FIG. 129 is a conceptual block diagram of an educational system fororthopedic surgical education to be used by a student without a liveteacher.

FIG. 130 is a conceptual block diagram of an educational system fororthopedic surgical education to be used by a teacher that is locatedremotely relative to students.

FIG. 131 is a conceptual block diagram of an educational system fororthopedic surgical education to be used by a teacher and a plurality ofstudents where one or more of the students are located remotely relativeto the teacher.

FIGS. 132 and 133 are conceptual block diagrams of other educationalsystems that use MR and/or VR for orthopedic surgical education.

FIG. 134 is a conceptual block diagram of an educational system thatuses MR and/or VR for orthopedic surgical education where the teacherand the students are able to manipulate different virtual models thatinclude virtual information.

FIG. 135 is a conceptual block diagram of an educational system that useMR and/or VR for orthopedic surgical education where the teacher is ableto assign manipulation rights to a virtual model to allow students tomanipulate the virtual model.

FIGS. 136-139 are flow diagrams illustrating educational techniques thatcan be performed with the aid of MR and/or VR.

FIG. 140 is a conceptual block diagram of an educational system that useMR and/or VR for orthopedic surgical education where a user is able tolaunch a manipulatable copy of a virtual model.

FIG. 141 is a conceptual block diagram of an educational system that useMR and/or VR for orthopedic surgical education where students andteachers are able to view and compare several different virtual models.

FIG. 142 is a conceptual block diagram of an educational system that useMR and/or VR for orthopedic surgical education where a teacher has avirtual control menu for controlling MR/VR educational content.

FIG. 143 is a conceptual block diagram of an educational system that useMR and/or VR for orthopedic surgical education where a teacher andstudents have virtual control elements for controlling MR/VR educationalcontent.

FIG. 144 is a flow diagram illustrating educational techniques that canbe performed with the aid of MR and/or VR.

FIG. 145 is a conceptual block diagram of an educational system thatincludes features to help educate a remote user on specific details ofan ongoing surgical procedure.

FIGS. 146 and 147 are flow diagrams illustrating inter-operativeeducational techniques that to help educate a remote user on specificdetails of an ongoing surgical procedure.

FIG. 148 is one conceptual example showing a virtual teacher model of anexample virtual shoulder and multiple student models that have similarvirtual content to the teacher model.

FIG. 149 is a flow diagram illustrating example techniques for MR aidedvalidation of anatomy preparation, in accordance with one or moretechniques of this disclosure.

FIG. 150 is a flowchart illustrating example stages of an ankle jointrepair surgery.

FIGS. 151A and 151B are conceptual diagrams illustrating exampleattachment of guide pins to a tibia.

FIG. 152 is a conceptual diagram illustrating example drilling of holesin a tibia.

FIG. 153 is a conceptual diagram illustrating example resection of atibia.

FIGS. 154A and 154B are conceptual diagrams illustrating example guidepins installed in a talus during a talus preparation process.

FIG. 155 is a conceptual diagram illustrating example resection of atalus.

FIG. 156 is a conceptual diagram of an example ankle after performanceof a tibial resection and a talar resection.

FIGS. 157A-157C are conceptual diagrams illustrating an example oftibial tray trialing.

FIG. 158 is a conceptual diagram illustrating an example creation oftibial implant anchorage.

FIGS. 159A and 159B are conceptual diagrams illustrating an exampleattachment of guide pins to a talus.

FIG. 160 is a conceptual diagram of an example talar resection guide ona talus.

FIG. 161 is a conceptual diagram of an example posterior talar chamferresection.

FIGS. 162 and 163 are conceptual diagrams of example anterior talarchamfer resections.

FIGS. 164 and 165 are conceptual diagrams illustrating an examplecreation of talar implant anchorage.

FIG. 166 is a conceptual diagram illustrating an example tibial implant.

FIG. 167 is a conceptual diagram illustrating an example of a preparedtibia.

FIG. 168 is a conceptual diagram illustrating example impaction of atibial implant into a tibia.

FIG. 169 is a conceptual diagram illustrating an example talar implant.

FIG. 170 is a conceptual diagram illustrating example impaction of atalar implant into a talus.

FIG. 171 is a conceptual diagram illustrating an example bearingimplanted between a tibial implant and a talar implant.

FIG. 172 is a flow diagram illustrating an example technique for MRaided surgery, in accordance with one or more techniques of thisdisclosure.

DETAILED DESCRIPTION

Certain examples of this disclosure are described with reference to theaccompanying drawings, wherein like reference numerals denote likeelements. It should be understood, however, that the accompanyingdrawings illustrate only the various implementations described hereinand are not meant to limit the scope of various technologies describedherein. The drawings show and describe various examples of thisdisclosure.

In the following description, numerous details are set forth to providean understanding of the present invention. However, it will beunderstood by those skilled in the art that the present invention may bepracticed without these details and that numerous variations ormodifications from the described examples may be possible.

Orthopedic surgery can involve implanting one or more prosthetic devicesto repair or replace a patient's damaged or diseased joint. Today,virtual surgical planning tools are available that use image data of thediseased or damaged joint to generate an accurate three-dimensional bonemodel that can be viewed and manipulated preoperatively by the surgeon.These tools can enhance surgical outcomes by allowing the surgeon tosimulate the surgery, select or design an implant that more closelymatches the contours of the patient's actual bone, and select or designsurgical instruments and guide tools that are adapted specifically forrepairing the bone of a particular patient. Use of these planning toolstypically results in generation of a preoperative surgical plan,complete with an implant and surgical instruments that are selected ormanufactured for the individual patient. Oftentimes, once in the actualoperating environment, the surgeon may desire to verify the preoperativesurgical plan intraoperatively relative to the patient's actual bone.This verification may result in a determination that an adjustment tothe preoperative surgical plan is needed, such as a different implant, adifferent positioning or orientation of the implant, and/or a differentsurgical guide for carrying out the surgical plan. In addition, asurgeon may want to view details of the preoperative surgical planrelative to the patient's real bone during the actual procedure in orderto more efficiently and accurately position and orient the implantcomponents. For example, the surgeon may want to obtain intra-operativevisualization that provides guidance for positioning and orientation ofimplant components, guidance for preparation of bone or tissue toreceive the implant components, guidance for reviewing the details of aprocedure or procedural step, and/or guidance for selection of tools orimplants and tracking of surgical procedure workflow.

Accordingly, this disclosure describes systems and methods for using amixed reality (MR) visualization system to assist with creation,implementation, verification, and/or modification of a surgical planbefore and during a surgical procedure. Because MR, or in some instancesVR, may be used to interact with the surgical plan, this disclosure mayalso refer to the surgical plan as a “virtual” surgical plan.Visualization tools other than or in addition to mixed realityvisualization systems may be used in accordance with techniques of thisdisclosure. A surgical plan, e.g., as generated by the BLUEPRINT™ systemor another surgical planning platform, may include information defininga variety of features of a surgical procedure, such as features ofparticular surgical procedure steps to be performed on a patient by asurgeon according to the surgical plan including, for example, bone ortissue preparation steps and/or steps for selection, modification and/orplacement of implant components. Such information may include, invarious examples, dimensions, shapes, angles, surface contours, and/ororientations of implant components to be selected or modified bysurgeons, dimensions, shapes, angles, surface contours and/ororientations to be defined in bone or tissue by the surgeon in bone ortissue preparation steps, and/or positions, axes, planes, angle and/orentry points defining placement of implant components by the surgeonrelative to patient bone or tissue. Information such as dimensions,shapes, angles, surface contours, and/or orientations of anatomicalfeatures of the patient may be derived from imaging (e.g., x-ray, CT,MRI, ultrasound or other images), direct observation, or othertechniques.

In this disclosure, the term “mixed reality” (MR) refers to thepresentation of virtual objects such that a user sees images thatinclude both real, physical objects and virtual objects. Virtual objectsmay include text, 2-dimensional surfaces, 3-dimensional models, or otheruser-perceptible elements that are not actually present in the physical,real-world environment in which they are presented as coexisting. Inaddition, virtual objects described in various examples of thisdisclosure may include graphics, images, animations or videos, e.g.,presented as 3D virtual objects or 2D virtual objects. Virtual objectsmay also be referred to as virtual elements. Such elements may or maynot be analogs of real-world objects. In some examples, in mixedreality, a camera may capture images of the real world and modify theimages to present virtual objects in the context of the real world. Insuch examples, the modified images may be displayed on a screen, whichmay be head-mounted, handheld, or otherwise viewable by a user. Thistype of mixed reality is increasingly common on smartphones, such aswhere a user can point a smartphone's camera at a sign written in aforeign language and see in the smartphone's screen a translation in theuser's own language of the sign superimposed on the sign along with therest of the scene captured by the camera. In some examples, in mixedreality, see-through (e.g., transparent) holographic lenses, which maybe referred to as waveguides, may permit the user to view real-worldobjects, i.e., actual objects in a real-world environment, such as realanatomy, through the holographic lenses and also concurrently viewvirtual objects.

The Microsoft HOLOLENS™ headset, available from Microsoft Corporation ofRedmond, Wash., is an example of a MR device that includes see-throughholographic lenses, sometimes referred to as waveguides, that permit auser to view real-world objects through the lens and concurrently viewprojected 3D holographic objects. The Microsoft HOLOLENS™ headset, orsimilar waveguide-based visualization devices, are examples of an MRvisualization device that may be used in accordance with some examplesof this disclosure. Some holographic lenses may present holographicobjects with some degree of transparency through see-through holographiclenses so that the user views real-world objects and virtual,holographic objects. In some examples, some holographic lenses may, attimes, completely prevent the user from viewing real-world objects andinstead may allow the user to view entirely virtual environments. Theterm mixed reality may also encompass scenarios where one or more usersare able to perceive one or more virtual objects generated byholographic projection. In other words, “mixed reality” may encompassthe case where a holographic projector generates holograms of elementsthat appear to a user to be present in the user's actual physicalenvironment.

In some examples, in mixed reality, the positions of some or allpresented virtual objects are related to positions of physical objectsin the real world. For example, a virtual object may be tethered to atable in the real world, such that the user can see the virtual objectwhen the user looks in the direction of the table but does not see thevirtual object when the table is not in the user's field of view. Insome examples, in mixed reality, the positions of some or all presentedvirtual objects are unrelated to positions of physical objects in thereal world. For instance, a virtual item may always appear in the topright of the user's field of vision, regardless of where the user islooking.

Augmented reality (AR) is similar to MR in the presentation of bothreal-world and virtual elements, but AR generally refers topresentations that are mostly real, with a few virtual additions to“augment” the real-world presentation. For purposes of this disclosure,MR is considered to include AR. For example, in AR, parts of the user'sphysical environment that are in shadow can be selectively brightenedwithout brightening other areas of the user's physical environment. Thisexample is also an instance of MR in that the selectively-brightenedareas may be considered virtual objects superimposed on the parts of theuser's physical environment that are in shadow.

Furthermore, in this disclosure, the term “virtual reality” (VR) refersto an immersive artificial environment that a user experiences throughsensory stimuli (such as sights and sounds) provided by a computer.Thus, in virtual reality, the user may not see any physical objects asthey exist in the real world. Video games set in imaginary worlds are acommon example of VR. The term “VR” also encompasses scenarios where theuser is presented with a fully artificial environment in which somevirtual object's locations are based on the locations of correspondingphysical objects as they relate to the user. Walk-through VR attractionsare examples of this type of VR.

The term “extended reality” (XR) is a term that encompasses a spectrumof user experiences that includes virtual reality, mixed reality,augmented reality, and other user experiences that involve thepresentation of at least some perceptible elements as existing in theuser's environment that are not present in the user's real-worldenvironment. Thus, the term “extended reality” may be considered a genusfor MR and VR. XR visualizations may be presented in any of thetechniques for presenting mixed reality discussed elsewhere in thisdisclosure or presented using techniques for presenting VR, such as VRgoggles.

These mixed reality systems and methods can be part of an intelligentsurgical planning system that includes multiple subsystems that can beused to enhance surgical outcomes. In addition to the preoperative andintraoperative applications discussed above, an intelligent surgicalplanning system can include postoperative tools to assist with patientrecovery and which can provide information that can be used to assistwith and plan future surgical revisions or surgical cases for otherpatients.

Accordingly, systems and methods are also described herein that can beincorporated into an intelligent surgical planning system, such asartificial intelligence systems to assist with planning, implants withembedded sensors (e.g., smart implants) to provide postoperativefeedback for use by the healthcare provider and the artificialintelligence system, and mobile applications to monitor and provideinformation to the patient and the healthcare provider in real-time ornear real-time.

Visualization tools are available that utilize patient image data togenerate three-dimensional models of bone contours to facilitatepreoperative planning for joint repairs and replacements. These toolsallow surgeons to design and/or select surgical guides and implantcomponents that closely match the patient's anatomy. These tools canimprove surgical outcomes by customizing a surgical plan for eachpatient. An example of such a visualization tool for shoulder repairs isthe BLUEPRINT™ system available from Wright Medical Technology, Inc. TheBLUEPRINT™ system provides the surgeon with two-dimensional planar viewsof the bone repair region as well as a three-dimensional virtual modelof the repair region. The surgeon can use the BLUEPRINT™ system toselect, design or modify appropriate implant components, determine howbest to position and orient the implant components and how to shape thesurface of the bone to receive the components, and design, select ormodify surgical guide tool(s) or instruments to carry out the surgicalplan. The information generated by the BLUEPRINT™ system is compiled ina preoperative surgical plan for the patient that is stored in adatabase at an appropriate location (e.g., on a server in a wide areanetwork, a local area network, or a global network) where it can beaccessed by the surgeon or other care provider, including before andduring the actual surgery.

FIG. 1 is a block diagram of an orthopedic surgical system 100 accordingto an example of this disclosure. Orthopedic surgical system 100includes a set of subsystems. In the example of FIG. 1, the subsystemsinclude a virtual planning system 102, a planning support system 104, amanufacturing and delivery system 106, an intraoperative guidance system108, a medical education system 110, a monitoring system 112, apredictive analytics system 114, and a communications network 116. Inother examples, orthopedic surgical system 100 may include more, fewer,or different subsystems. For example, orthopedic surgical system 100 mayomit medical education system 110, monitoring system 112, predictiveanalytics system 114, and/or other subsystems. In some examples,orthopedic surgical system 100 may be used for surgical tracking, inwhich case orthopedic surgical system 100 may be referred to as asurgical tracking system. In other cases, orthopedic surgical system 100may be generally referred to as a medical device system.

Users of orthopedic surgical system 100 may use virtual planning system102 to plan orthopedic surgeries. Users of orthopedic surgical system100 may use planning support system 104 to review surgical plansgenerated using orthopedic surgical system 100. Manufacturing anddelivery system 106 may assist with the manufacture and delivery ofitems needed to perform orthopedic surgeries. Intraoperative guidancesystem 108 provides guidance to assist users of orthopedic surgicalsystem 100 in performing orthopedic surgeries. Medical education system110 may assist with the education of users, such as healthcareprofessionals, patients, and other types of individuals. Pre- andpostoperative monitoring system 112 may assist with monitoring patientsbefore and after the patients undergo surgery. Predictive analyticssystem 114 may assist healthcare professionals with various types ofpredictions. For example, predictive analytics system 114 may applyartificial intelligence techniques to determine a classification of acondition of an orthopedic joint, e.g., a diagnosis, determine whichtype of surgery to perform on a patient and/or which type of implant tobe used in the procedure, determine types of items that may be neededduring the surgery, and so on.

The subsystems of orthopedic surgical system 100 (i.e., virtual planningsystem 102, planning support system 104, manufacturing and deliverysystem 106, intraoperative guidance system 108, medical education system110, pre- and postoperative monitoring system 112, and predictiveanalytics system 114) may include various systems. The systems in thesubsystems of orthopedic surgical system 100 may include various typesof computing systems, computing devices, including server computers,personal computers, tablet computers, smartphones, display devices,Internet of Things (IoT) devices, visualization devices (e.g., mixedreality (MR) visualization devices, virtual reality (VR) visualizationdevices, holographic projectors, or other devices for presentingextended reality (XR) visualizations), surgical tools, and so on. Aholographic projector, in some examples, may project a hologram forgeneral viewing by multiple users or a single user without a headset,rather than viewing only by a user wearing a headset. For example,virtual planning system 102 may include a MR visualization device andone or more server devices, planning support system 104 may include oneor more personal computers and one or more server devices, and so on. Acomputing system is a set of one or more computing systems configured tooperate as a system. In some examples, one or more devices may be sharedbetween two or more of the subsystems of orthopedic surgical system 100.For instance, in the previous examples, virtual planning system 102 andplanning support system 104 may include the same server devices.

In the example of FIG. 1, the devices included in the subsystems oforthopedic surgical system 100 may communicate using communicationsnetwork 116. Communications network 116 may include various types ofcommunication networks including one or more wide-area networks, such asthe Internet, local area networks, and so on. In some examples,communications network 116 may include wired and/or wirelesscommunication links.

Many variations of orthopedic surgical system 100 are possible inaccordance with techniques of this disclosure. Such variations mayinclude more or fewer subsystems than the version of orthopedic surgicalsystem 100 shown in FIG. 1. For example, FIG. 2 is a block diagram of anorthopedic surgical system 200 that includes one or more mixed reality(MR) systems, according to an example of this disclosure. Orthopedicsurgical system 200 may be used for creating, verifying, updating,modifying and/or implementing a surgical plan. In some examples, thesurgical plan can be created preoperatively, such as by using a virtualsurgical planning system (e.g., the BLUEPRINT™ system), and thenverified, modified, updated, and viewed intraoperatively, e.g., using MRvisualization of the surgical plan. In other examples, orthopedicsurgical system 200 can be used to create the surgical plan immediatelyprior to surgery or intraoperatively, as needed. In some examples,orthopedic surgical system 200 may be used for surgical tracking, inwhich case orthopedic surgical system 200 may be referred to as asurgical tracking system. In other cases, orthopedic surgical system 200may be generally referred to as a medical device system.

In the example of FIG. 2, orthopedic surgical system 200 includes apreoperative surgical planning system 202, a healthcare facility 204(e.g., a surgical center or hospital), a storage system 206, and anetwork 208 that allows a user at healthcare facility 204 to accessstored patient information, such as medical history, image datacorresponding to the damaged joint or bone and various parameterscorresponding to a surgical plan that has been created preoperatively(as examples). Preoperative surgical planning system 202 may beequivalent to virtual planning system 102 of FIG. 1 and, in someexamples, may generally correspond to a virtual planning system similaror identical to the BLUEPRINT™ system.

In the example of FIG. 2, healthcare facility 204 includes a mixedreality (MR) system 212. In some examples of this disclosure, MR system212 includes one or more processing device(s) (P) 210 to providefunctionalities that will be described in further detail below.Processing device(s) 210 may also be referred to as processor(s). Inaddition, one or more users of MR system 212 (e.g., a surgeon, nurse, orother care provider) can use processing device(s) (P) 210 to generate arequest for a particular surgical plan or other patient information thatis transmitted to storage system 206 via network 208. In response,storage system 206 returns the requested patient information to MRsystem 212. In some examples, the users can use other processingdevice(s) to request and receive information, such as one or moreprocessing devices that are part of MR system 212, but not part of anyvisualization device, or one or more processing devices that are part ofa visualization device (e.g., visualization device 213) of MR system212, or a combination of one or more processing devices that are part ofMR system 212, but not part of any visualization device, and one or moreprocessing devices that are part of a visualization device (e.g.,visualization device 213) that is part of MR system 212.

In some examples, multiple users can simultaneously use MR system 212.For example, MR system 212 can be used in a spectator mode in whichmultiple users each use their own visualization devices so that theusers can view the same information at the same time and from the samepoint of view. In some examples, MR system 212 may be used in a mode inwhich multiple users each use their own visualization devices so thatthe users can view the same information from different points of view.

In some examples, processing device(s) 210 can provide a user interfaceto display data and receive input from users at healthcare facility 204.Processing device(s) 210 may be configured to control visualizationdevice 213 to present a user interface. Furthermore, processingdevice(s) 210 may be configured to control visualization device 213 topresent virtual images, such as 3D virtual models, 2D images, and so on.Processing device(s) 210 can include a variety of different processingor computing devices, such as servers, desktop computers, laptopcomputers, tablets, mobile phones and other electronic computingdevices, or processors within such devices. In some examples, one ormore of processing device(s) 210 can be located remote from healthcarefacility 204. In some examples, processing device(s) 210 reside withinvisualization device 213. In some examples, at least one of processingdevice(s) 210 is external to visualization device 213. In some examples,one or more processing device(s) 210 reside within visualization device213 and one or more of processing device(s) 210 are external tovisualization device 213.

In the example of FIG. 2, MR system 212 also includes one or more memoryor storage device(s) (M) 215 for storing data and instructions ofsoftware that can be executed by processing device(s) 210. Theinstructions of software can correspond to the functionality of MRsystem 212 described herein. In some examples, the functionalities of avirtual surgical planning application, such as the BLUEPRINT™ system,can also be stored and executed by processing device(s) 210 inconjunction with memory storage device(s) (M) 215. For instance, memoryor storage system 215 may be configured to store data corresponding toat least a portion of a virtual surgical plan. In some examples, storagesystem 206 may be configured to store data corresponding to at least aportion of a virtual surgical plan. In some examples, memory or storagedevice(s) (M) 215 reside within visualization device 213. In someexamples, memory or storage device(s) (M) 215 are external tovisualization device 213. In some examples, memory or storage device(s)(M) 215 include a combination of one or more memory or storage deviceswithin visualization device 213 and one or more memory or storagedevices external to the visualization device.

Network 208 may be equivalent to network 116. Network 208 can includeone or more wide area networks, local area networks, and/or globalnetworks (e.g., the Internet) that connect preoperative surgicalplanning system 202 and MR system 212 to storage system 206. Storagesystem 206 can include one or more databases that can contain patientinformation, medical information, patient image data, and parametersthat define the surgical plans. For example, medical images of thepatient's diseased or damaged bone typically are generatedpreoperatively in preparation for an orthopedic surgical procedure. Themedical images can include images of the relevant bone(s) taken alongthe sagittal plane and the coronal plane of the patient's body. Themedical images can include X-ray images, magnetic resonance imaging(MRI) images, computerized tomography (CT) images, ultrasound images,and/or any other type of 2D or 3D image that provides information aboutthe relevant surgical area. Storage system 206 also can include dataidentifying the implant components selected for a particular patient(e.g., type, size, etc.), surgical guides selected for a particularpatient, and details of the surgical procedure, such as entry points,cutting planes, drilling axes, reaming depths, etc. Storage system 206can be a cloud-based storage system (as shown) or can be located athealthcare facility 204 or at the location of preoperative surgicalplanning system 202 or can be part of MR system 212 or visualizationdevice (VD) 213, as examples.

MR system 212 can be used by a surgeon before (e.g., preoperatively) orduring the surgical procedure (e.g., intraoperatively) to create,review, verify, update, modify and/or implement a surgical plan. In someexamples, MR system 212 may also be used after the surgical procedure(e.g., postoperatively) to review the results of the surgical procedure,assess whether revisions are required, or perform other postoperativetasks. To that end, MR system 212 may include a visualization device 213that may be worn by the surgeon and (as will be explained in furtherdetail below) is operable to display a variety of types of information,including a 3D virtual image of the patient's diseased, damaged, orpostsurgical joint and details of the surgical plan, such as a 3Dvirtual image of the prosthetic implant components selected for thesurgical plan, 3D virtual images of entry points for positioning theprosthetic components, alignment axes and cutting planes for aligningcutting or reaming tools to shape the bone surfaces, or drilling toolsto define one or more holes in the bone surfaces, in the surgicalprocedure to properly orient and position the prosthetic components,surgical guides and instruments and their placement on the damagedjoint, and any other information that may be useful to the surgeon toimplement the surgical plan. MR system 212 can generate images of thisinformation that are perceptible to the user of the visualization device213 before and/or during the surgical procedure.

In some examples, MR system 212 includes multiple visualization devices(e.g., multiple instances of visualization device 213) so that multipleusers can simultaneously see the same images and share the same 3Dscene. In some such examples, one of the visualization devices can bedesignated as the master device and the other visualization devices canbe designated as observers or spectators. Any observer device can bere-designated as the master device at any time, as may be desired by theusers of MR system 212.

In this way, FIG. 2 illustrates a surgical planning system that includesa preoperative surgical planning system 202 to generate a virtualsurgical plan customized to repair an anatomy of interest of aparticular patient. For example, the virtual surgical plan may include aplan for an orthopedic joint repair surgical procedure (e.g., to attacha prosthetic to anatomy of a patient), such as one of a standard totalshoulder arthroplasty or a reverse shoulder arthroplasty. In thisexample, details of the virtual surgical plan may include detailsrelating to at least one of preparation of anatomy for attachment of aprosthetic or attachment of the prosthetic to the anatomy. For instance,details of the virtual surgical plan may include details relating to atleast one of preparation of a glenoid bone, preparation of a humeralbone, attachment of a prosthetic to the glenoid bone, or attachment of aprosthetic to the humeral bone. In some examples, the orthopedic jointrepair surgical procedure is one of a stemless standard total shoulderarthroplasty, a stemmed standard total shoulder arthroplasty, a stemlessreverse shoulder arthroplasty, a stemmed reverse shoulder arthroplasty,an augmented glenoid standard total shoulder arthroplasty, and anaugmented glenoid reverse shoulder arthroplasty.

The virtual surgical plan may include a 3D virtual model correspondingto the anatomy of interest of the particular patient and a 3D model of aprosthetic component matched to the particular patient to repair theanatomy of interest or selected to repair the anatomy of interest.Furthermore, in the example of FIG. 2, the surgical planning systemincludes a storage system 206 to store data corresponding to the virtualsurgical plan. The surgical planning system of FIG. 2 also includes MRsystem 212, which may comprise visualization device 213. In someexamples, visualization device 213 is wearable by a user. In someexamples, visualization device 213 is held by a user, or rests on asurface in a place accessible to the user. MR system 212 may beconfigured to present a user interface via visualization device 213. Theuser interface may present details of the virtual surgical plan for aparticular patient. For instance, the details of the virtual surgicalplan may include a 3D virtual model of an anatomy of interest of theparticular patient. The user interface is visually perceptible to theuser when the user is using visualization device 213. For instance, inone example, a screen of visualization device 213 may display real-worldimages and the user interface on a screen. In some examples,visualization device 213 may project virtual, holographic images ontosee-through holographic lenses and also permit a user to see real-worldobjects of a real-world environment through the lenses. In other words,visualization device 213 may comprise one or more see-throughholographic lenses and one or more display devices that present imageryto the user via the holographic lenses to present the user interface tothe user.

In some examples, visualization device 213 is configured such that theuser can manipulate the user interface (which is visually perceptible tothe user when the user is wearing or otherwise using visualizationdevice 213) to request and view details of the virtual surgical plan forthe particular patient, including a 3D virtual model of the anatomy ofinterest (e.g., a 3D virtual bone model of the anatomy of interest, suchas a glenoid bone or a humeral bone) and/or a 3D model of the prostheticcomponent selected to repair an anatomy of interest. In some suchexamples, visualization device 213 is configured such that the user canmanipulate the user interface so that the user can view the virtualsurgical plan intraoperatively, including (at least in some examples)the 3D virtual model of the anatomy of interest (e.g., a 3D virtual bonemodel of the anatomy of interest). In some examples, MR system 212 canbe operated in an augmented surgery mode in which the user canmanipulate the user interface intraoperatively so that the user canvisually perceive details of the virtual surgical plan projected in areal environment, e.g., on a real anatomy of interest of the particularpatient. In this disclosure, the terms real and real world may be usedin a similar manner. For example, MR system 212 may present one or morevirtual objects that provide guidance for preparation of a bone surfaceand placement of a prosthetic implant on the bone surface. Visualizationdevice 213 may present one or more virtual objects in a manner in whichthe virtual objects appear to be overlaid on an actual, real anatomicalobject of the patient, within a real-world environment, e.g., bydisplaying the virtual object(s) with actual, real-world patient anatomyviewed by the user through holographic lenses. For example, the virtualobjects may be 3D virtual objects that appear to reside within thereal-world environment with the actual, real anatomical object.

FIG. 3 is a flowchart illustrating example phases of a surgicallifecycle 300. In the example of FIG. 3, surgical lifecycle 300 beginswith a preoperative phase (302). During the preoperative phase, asurgical plan is developed. The preoperative phase is followed by amanufacturing and delivery phase (304). During the manufacturing anddelivery phase, patient-specific items, such as parts and equipment,needed for executing the surgical plan are manufactured and delivered toa surgical site. In some examples, it is unnecessary to manufacturepatient-specific items in order to execute the surgical plan. Anintraoperative phase follows the manufacturing and delivery phase (306).The surgical plan is executed during the intraoperative phase. In otherwords, one or more persons perform the surgery on the patient during theintraoperative phase. The intraoperative phase is followed by thepostoperative phase (308). The postoperative phase includes activitiesoccurring after the surgical plan is complete. For example, the patientmay be monitored during the postoperative phase for complications.

As described in this disclosure, orthopedic surgical system 100 (FIG. 1)may be used in one or more of preoperative phase 302, the manufacturingand delivery phase 304, the intraoperative phase 306, and thepostoperative phase 308. For example, virtual planning system 102 andplanning support system 104 may be used in preoperative phase 302.Manufacturing and delivery system 106 may be used in the manufacturingand delivery phase 304. Intraoperative guidance system 108 may be usedin intraoperative phase 306. Some of the systems of FIG. 1 may be usedin multiple phases of FIG. 3. For example, medical education system 110may be used in one or more of preoperative phase 302, intraoperativephase 306, and postoperative phase 308; pre- and postoperativemonitoring system 112 may be used in preoperative phase 302 andpostoperative phase 308. Predictive analytics system 114 may be used inpreoperative phase 302 and postoperative phase 308.

Various workflows may exist within the surgical process of FIG. 3. Forexample, different workflows within the surgical process of FIG. 3 maybe appropriate for different types of surgeries. FIG. 4 is a flowchartillustrating preoperative, intraoperative and postoperative workflows insupport of an orthopedic surgical procedure. In the example of FIG. 4,the surgical process begins with a medical consultation (400). Duringthe medical consultation (400), a healthcare professional evaluates amedical condition of a patient. For instance, the healthcareprofessional may consult the patient with respect to the patient'ssymptoms. During the medical consultation (400), the healthcareprofessional may also discuss various treatment options with thepatient. For instance, the healthcare professional may describe one ormore different surgeries to address the patient's symptoms.

Furthermore, the example of FIG. 4 includes a case creation step (402).In other examples, the case creation step occurs before the medicalconsultation step. During the case creation step, the medicalprofessional or other user establishes an electronic case file for thepatient. The electronic case file for the patient may includeinformation related to the patient, such as data regarding the patient'ssymptoms, patient range of motion observations, data regarding asurgical plan for the patient, medical images of the patients, notesregarding the patient, billing information regarding the patient, and soon.

The example of FIG. 4 includes a preoperative patient monitoring phase(404). During the preoperative patient monitoring phase, the patient'ssymptoms may be monitored. For example, the patient may be sufferingfrom pain associated with arthritis in the patient's shoulder. In thisexample, the patient's symptoms may not yet rise to the level ofrequiring an arthroplasty to replace the patient's shoulder. However,arthritis typically worsens over time. Accordingly, the patient'ssymptoms may be monitored to determine whether the time has come toperform a surgery on the patient's shoulder. Observations from thepreoperative patient monitoring phase may be stored in the electroniccase file for the patient. In some examples, predictive analytics system114 may be used to predict when the patient may need surgery, to predicta course of treatment to delay or avoid surgery or make otherpredictions with respect to the patient's health.

Additionally, in the example of FIG. 4, a medical image acquisition stepoccurs during the preoperative phase (406). During the image acquisitionstep, medical images of the patient are generated. The medical imagesmay be generated in a variety of ways. For instance, the images may begenerated using a Computed Tomography (CT) process, a Magnetic ResonanceImaging (MRI) process, an ultrasound process, or another imagingprocess. The medical images generated during the image acquisition stepinclude images of an anatomy of interest of the patient. For instance,if the patient's symptoms involve the patient's shoulder, medical imagesof the patient's shoulder may be generated. The medical images may beadded to the patient's electronic case file. Healthcare professionalsmay be able to use the medical images in one or more of thepreoperative, intraoperative, and postoperative phases.

Furthermore, in the example of FIG. 4, an automatic processing step mayoccur (408). During the automatic processing step, virtual planningsystem 102 (FIG. 1) may automatically develop a preliminary surgicalplan for the patient. In some examples of this disclosure, virtualplanning system 102 may use machine learning techniques to develop thepreliminary surgical plan based on information in the patient's virtualcase file.

The example of FIG. 4 also includes a manual correction step (410).During the manual correction step, one or more human users may check andcorrect the determinations made during the automatic processing step. Insome examples of this disclosure, one or more users may use mixedreality or virtual reality visualization devices during the manualcorrection step. In some examples, changes made during the manualcorrection step may be used as training data to refine the machinelearning techniques applied by virtual planning system 102 during theautomatic processing step.

A virtual planning step (412) may follow the manual correction step inFIG. 4. During the virtual planning step, a healthcare professional maydevelop a surgical plan for the patient. In some examples of thisdisclosure, one or more users may use mixed reality or virtual realityvisualization devices during development of the surgical plan for thepatient.

Furthermore, in the example of FIG. 4, intraoperative guidance may begenerated (414). The intraoperative guidance may include guidance to asurgeon on how to execute the surgical plan. In some examples of thisdisclosure, virtual planning system 102 may generate at least part ofthe intraoperative guidance. In some examples, the surgeon or other usermay contribute to the intraoperative guidance.

Additionally, in the example of FIG. 4, a step of selecting andmanufacturing surgical items is performed (416). During the step ofselecting and manufacturing surgical items, manufacturing and deliverysystem 106 (FIG. 1) may manufacture surgical items for use during thesurgery described by the surgical plan. For example, the surgical itemsmay include surgical implants, surgical tools, and other items requiredto perform the surgery described by the surgical plan.

In the example of FIG. 4, a surgical procedure may be performed withguidance from intraoperative system 108 (FIG. 1) (418). For example, asurgeon may perform the surgery while wearing a head-mounted MRvisualization device of intraoperative system 108 that presents guidanceinformation to the surgeon. The guidance information may help guide thesurgeon through the surgery, providing guidance for various steps in asurgical workflow, including sequence of steps, details of individualsteps, and tool or implant selection, implant placement and position,and bone surface preparation for various steps in the surgical procedureworkflow.

Postoperative patient monitoring may occur after completion of thesurgical procedure (420). During the postoperative patient monitoringstep, healthcare outcomes of the patient may be monitored. Healthcareoutcomes may include relief from symptoms, ranges of motion,complications, performance of implanted surgical items, and so on. Pre-and postoperative monitoring system 112 (FIG. 1) may assist in thepostoperative patient monitoring step.

The medical consultation, case creation, preoperative patientmonitoring, image acquisition, automatic processing, manual correction,and virtual planning steps of FIG. 4 are part of preoperative phase 302of FIG. 3. The surgical procedures with guidance steps of FIG. 4 is partof intraoperative phase 306 of FIG. 3. The postoperative patientmonitoring step of FIG. 4 is part of postoperative phase 308 of FIG. 3.

As mentioned above, one or more of the subsystems of orthopedic surgicalsystem 100 may include one or more mixed reality (MR) systems, such asMR system 212 (FIG. 2). Each MR system may include a visualizationdevice. For instance, in the example of FIG. 2, MR system 212 includesvisualization device 213. In some examples, in addition to including avisualization device, an MR system may include external computingresources that support the operations of the visualization device. Forinstance, the visualization device of an MR system may becommunicatively coupled to a computing device (e.g., a personalcomputer, backpack computer, smartphone, etc.) that provides theexternal computing resources. Alternatively, adequate computingresources may be provided on or within visualization device 213 toperform necessary functions of the visualization device.

FIG. 5 is a schematic representation of visualization device 213 for usein an MR system, such as MR system 212 of FIG. 2, according to anexample of this disclosure. As shown in the example of FIG. 5,visualization device 213 can include a variety of electronic componentsfound in a computing system, including one or more processor(s) 514(e.g., microprocessors or other types of processing units) and memory516 that may be mounted on or within a frame 518. Furthermore, in theexample of FIG. 5, visualization device 213 may include a transparentscreen 520 that is positioned at eye level when visualization device 213is worn by a user. In some examples, screen 520 can include one or moreliquid crystal displays (LCDs) or other types of display screens onwhich images are perceptible to a surgeon who is wearing or otherwiseusing visualization device 213 via screen 520. Other display examplesinclude organic light emitting diode (OLED) displays. In some examples,visualization device 213 can operate to project 3D images onto theuser's retinas using techniques known in the art.

In some examples, screen 520 may include see-through holographic lenses.sometimes referred to as waveguides, that permit a user to seereal-world objects through (e.g., beyond) the lenses and also seeholographic imagery projected into the lenses and onto the user'sretinas by displays, such as liquid crystal on silicon (LCoS) displaydevices, which are sometimes referred to as light engines or projectors,operating as an example of a holographic projection system 538 withinvisualization device 213. In other words, visualization device 213 mayinclude one or more see-through holographic lenses to present virtualimages to a user. Hence, in some examples, visualization device 213 canoperate to project 3D images onto the user's retinas via screen 520,e.g., formed by holographic lenses. In this manner, visualization device213 may be configured to present a 3D virtual image to a user within areal-world view observed through screen 520, e.g., such that the virtualimage appears to form part of the real-world environment. In someexamples, visualization device 213 may be a Microsoft HOLOLENS™ headset,available from Microsoft Corporation, of Redmond, Wash., USA, or asimilar device, such as, for example, a similar MR visualization devicethat includes waveguides. The HOLOLENS™ device can be used to present 3Dvirtual objects via holographic lenses, or waveguides, while permittinga user to view actual objects in a real-world scene, i.e., in areal-world environment, through the holographic lenses.

Although the example of FIG. 5 illustrates visualization device 213 as ahead-wearable device, visualization device 213 may have other forms andform factors. For instance, in some examples, visualization device 213may be a handheld smartphone or tablet.

Visualization device 213 can also generate a user interface (UI) 522that is visible to the user, e.g., as holographic imagery projected intosee-through holographic lenses as described above. For example, UI 522can include a variety of selectable widgets 524 that allow the user tointeract with a mixed reality (MR) system, such as MR system 212 of FIG.2. Imagery presented by visualization device 213 may include, forexample, one or more 3D virtual objects. Details of an example of UI 522are described elsewhere in this disclosure. Visualization device 213also can include a speaker or other sensory devices 526 that may bepositioned adjacent the user's ears. Sensory devices 526 can conveyaudible information or other perceptible information (e.g., vibrations)to assist the user of visualization device 213.

Visualization device 213 can also include a transceiver 528 to connectvisualization device 213 to a processing device 510 and/or to network208 and/or to a computing cloud, such as via a wired communicationprotocol or a wireless protocol, e.g., Wi-Fi, Bluetooth, etc.Visualization device 213 also includes a variety of sensors to collectsensor data, such as one or more optical camera(s) 530 (or other opticalsensors) and one or more depth camera(s) 532 (or other depth sensors),mounted to, on or within frame 518. In some examples, the opticalsensor(s) 530 are operable to scan the geometry of the physicalenvironment in which a user of MR system 212 is located (e.g., anoperating room) and collect two-dimensional (2D) optical image data(either monochrome or color). Depth sensor(s) 532 are operable toprovide 3D image data, such as by employing time of flight, stereo orother known or future-developed techniques for determining depth andthereby generating image data in three dimensions. Other sensors caninclude motion sensors 533 (e.g., Inertial Mass Unit (IMU) sensors,accelerometers, etc.) to assist with tracking movement.

MR system 212 processes the sensor data so that geometric,environmental, textural, or other types of landmarks (e.g., corners,edges or other lines, walls, floors, objects) in the user's environmentor “scene” can be defined and movements within the scene can bedetected. As an example, the various types of sensor data can becombined or fused so that the user of visualization device 213 canperceive 3D images that can be positioned, or fixed and/or moved withinthe scene. When a 3D image is fixed in the scene, the user can walkaround the 3D image, view the 3D image from different perspectives, andmanipulate the 3D image within the scene using hand gestures, voicecommands, gaze line (or direction) and/or other control inputs. Asanother example, the sensor data can be processed so that the user canposition a 3D virtual object (e.g., a bone model) on an observedphysical object in the scene (e.g., a surface, the patient's real bone,etc.) and/or orient the 3D virtual object with other virtual imagesdisplayed in the scene. In some examples, the sensor data can beprocessed so that the user can position and fix a virtual representationof the surgical plan (or other widget, image or information) onto asurface, such as a wall of the operating room. Yet further, in someexamples, the sensor data can be used to recognize surgical instrumentsand the position and/or location of those instruments.

Visualization device 213 may include one or more processors 514 andmemory 516, e.g., within frame 518 of the visualization device. In someexamples, one or more external computing resources 536 process and storeinformation, such as sensor data, instead of or in addition to in-frameprocessor(s) 514 and memory 516. In this way, data processing andstorage may be performed by one or more processors 514 and memory 516within visualization device 213 and/or some of the processing andstorage requirements may be offloaded from visualization device 213.Hence, in some examples, one or more processors that control theoperation of visualization device 213 may be within visualization device213, e.g., as processor(s) 514. Alternatively, in some examples, atleast one of the processors that controls the operation of visualizationdevice 213 may be external to visualization device 213, e.g., asprocessor(s) 210. Likewise, operation of visualization device 213 may,in some examples, be controlled in part by a combination one or moreprocessors 514 within the visualization device and one or moreprocessors 210 external to visualization device 213.

For instance, in some examples, when visualization device 213 is in thecontext of FIG. 2, processing of the sensor data can be performed byprocessing device(s) 210 in conjunction with memory or storage device(s)(M) 215. In some examples, processor(s) 514 and memory 516 mounted toframe 518 may provide sufficient computing resources to process thesensor data collected by cameras 530, 532 and motion sensors 533. Insome examples, the sensor data can be processed using a SimultaneousLocalization and Mapping (SLAM) algorithm, or other known orfuture-developed algorithms for processing and mapping 2D and 3D imagedata and tracking the position of visualization device 213 in the 3Dscene. In some examples, image tracking may be performed using sensorprocessing and tracking functionality provided by the MicrosoftHOLOLENS™ system, e.g., by one or more sensors and processors 514 withina visualization device 213 substantially conforming to the MicrosoftHOLOLENS™ device or a similar mixed reality (MR) visualization device.

In some examples, MR system 212 can also include user-operated controldevice(s) 534 that allow the user to operate MR system 212, use MRsystem 212 in spectator mode (either as master or observer), interactwith UI 522 and/or otherwise provide commands or requests to processingdevice(s) 210 or other systems connected to network 208. As examples,control device(s) 534 can include a microphone, a touch pad, a controlpanel, a motion sensor or other types of control input devices withwhich the user can interact.

FIG. 6 is a block diagram illustrating example components ofvisualization device 213 for use in a MR system. In the example of FIG.6, visualization device 213 includes processors 514, a power supply 600,display device(s) 602, speakers 604, microphone(s) 606, input device(s)608, output device(s) 610, storage device(s) 612, sensor(s) 614, andcommunication devices 616. In the example of FIG. 6, sensor(s) 616 mayinclude depth sensor(s) 532, optical sensor(s) 530, motion sensor(s)533, and orientation sensor(s) 618. Optical sensor(s) 530 may includecameras, such as Red-Green-Blue (RGB) video cameras, infrared cameras,or other types of sensors that form images from light. Display device(s)602 may display imagery to present a user interface to the user.

Speakers 604, in some examples, may form part of sensory devices 526shown in FIG. 5. In some examples, display devices 602 may includescreen 520 shown in FIG. 5. For example, as discussed with reference toFIG. 5, display device(s) 602 may include see-through holographiclenses, in combination with projectors, that permit a user to seereal-world objects, in a real-world environment, through the lenses, andalso see virtual 3D holographic imagery projected into the lenses andonto the user's retinas, e.g., by a holographic projection system. Inthis example, virtual 3D holographic objects may appear to be placedwithin the real-world environment. In some examples, display devices 602include one or more display screens, such as LCD display screens, OLEDdisplay screens, and so on. The user interface may present virtualimages of details of the virtual surgical plan for a particular patient.

In some examples, a user may interact with and control visualizationdevice 213 in a variety of ways. For example, microphones 606, andassociated speech recognition processing circuitry or software, mayrecognize voice commands spoken by the user and, in response, performany of a variety of operations, such as selection, activation, ordeactivation of various functions associated with surgical planning,intra-operative guidance, or the like. As another example, one or morecameras or other optical sensors 530 of sensors 614 may detect andinterpret gestures to perform operations as described above. As afurther example, sensors 614 may sense gaze direction and performvarious operations as described elsewhere in this disclosure. In someexamples, input devices 608 may receive manual input from a user, e.g.,via a handheld controller including one or more buttons, a keypad, atouchscreen, joystick, trackball, and/or other manual input media, andperform, in response to the manual user input, various operations asdescribed above.

As discussed above, surgical lifecycle 300 may include a preoperativephase 302 (FIG. 3). One or more users may use orthopedic surgical system100 in preoperative phase 302. For instance, orthopedic surgical system100 may include virtual planning system 102 to help the one or moreusers generate a virtual surgical plan that may be customized to ananatomy of interest of a particular patient. As described herein, thevirtual surgical plan may include a 3-dimensional virtual model thatcorresponds to the anatomy of interest of the particular patient and a3-dimensional model of one or more prosthetic components matched to theparticular patient to repair the anatomy of interest or selected torepair the anatomy of interest. The virtual surgical plan also mayinclude a 3-dimensional virtual model of guidance information to guide asurgeon in performing the surgical procedure, e.g., in preparing bonesurfaces or tissue and placing implantable prosthetic hardware relativeto such bone surfaces or tissue.

FIG. 7 is a conceptual diagram illustrating an example setting in whicha set of users use MR systems of orthopedic surgical system 100 duringpreoperative phase 302. In the example of FIG. 7, a surgeon may use(e.g., wear) a visualization device (e.g., visualization device 213) ofa first MR system 700A (e.g., MR system 212). The visualization deviceof MR system 700A may present MR preoperative planning content 702 tothe surgeon during preoperative phase 302. As described in detailelsewhere in this disclosure, MR preoperative planning content 702 mayhelp the surgeon plan for a surgery.

Furthermore, in the example of FIG. 7, one or more other users may usevisualization devices of MR systems of orthopedic surgical system 100 toview MR preoperative planning content 702. For example, a patient mayuse a visualization device of a second MR system 700B duringpreoperative phase 302. The visualization device of MR system 700B maypresent MR preoperative planning content 702 to the patient. Forinstance, as described in detail elsewhere in this disclosure, MRpreoperative planning content 702 may include virtual 3D modelinformation to be presented using MR to help the patient understand oneor more of the patient's current condition and the surgery to beperformed on the patient.

In the example of FIG. 7, a nurse or other healthcare professional mayuse a visualization device of a third MR system 700C during preoperativephase 302. The visualization device of MR system 700C may present MRpreoperative planning content 702 to the nurse or other healthcareprofessional. For instance, in one example, MR preoperative planningcontent 702 may help the nurse understand a surgery before the surgeryhappens.

Furthermore, in the example of FIG. 7, a second surgeon may use avisualization device of a fourth MR system 700D. The visualizationdevice of MR system 700D may present MR preoperative planning content702 to the second surgeon. This may allow the surgeons to collaborate todevelop and review a surgical plan for the patient. For instance,surgeons may view and manipulate the same preoperative planning content702 at the same or different times. MR systems 700A, 700B, 700C, and700D may collectively be referred to herein as “MR systems 700.”

Thus, as described in the examples above, two or more of the individualsdescribed above (e.g., the first surgeon, the patient, the nurse, andthe second surgeon) can view the same or different MR preoperativeplanning content 702 at the same time. In examples where two or more ofthe individuals are viewing the same MR preoperative planning content702 at the same time, the two or more individuals may concurrently viewthe same MR preoperative guidance content 702 from the same or differentperspectives. Moreover, in some examples, two or more of the individualsdescribed above can view the same or different MR preoperative planningcontent 702 at different times. Preoperative planning content 702 mayinclude an information model of a surgical plan, virtual 3D modelinformation representing patient anatomy, such as bone and/or tissue,alone, or in combination with virtual 3D model information representingsurgical procedure steps and/or implant placement and positioning.Examples of preoperative planning content 702 may include a surgicalplan for a shoulder arthroplasty, virtual 3D model informationrepresenting scapula and/or glenoid bone, or representing humeral bone,with virtual 3D model information of instruments to be applied to thebone or implants to be positioned on or in the bone. In some examples,multiple users may be able to change and manipulate preoperativeplanning content 702.

FIG. 8 is a flowchart illustrating example steps in preoperative phase302 of surgical lifecycle 300. In other examples, preoperative phase 302may include more, fewer, or different steps. Moreover, in otherexamples, one or more of the steps of FIG. 8 may be performed indifferent orders. In some examples, one or more of the steps may beperformed automatically within a surgical planning system such asvirtual planning system 102 (FIG. 1) or 202 (FIG. 2).

In the example of FIG. 8, a model of the area of interest is generated(800). For example, a scan (e.g., a CT scan, MRI scan, or other type ofscan) of the area of interest may be performed. For example, if the areaof interest is the patient's shoulder, a scan of the patient's shouldermay be performed. Furthermore, a pathology in the area of interest maybe classified (802). In some examples, the pathology of the area ofinterest may be classified based on the scan of the area of interest.For example, if the area of interest is the user's shoulder, a surgeonmay determine what is wrong with the patient's shoulder based on thescan of the patient's shoulder and provide a shoulder classificationindicating the classification or diagnosis, e.g., such as primaryglenoid humeral osteoarthritis (PGHOA), rotator cuff tear arthropathy(RCTA) instability, massive rotator cuff tear (MRCT), rheumatoidarthritis, post-traumatic arthritis, and osteoarthritis.

Additionally, a surgical plan may be selected based on the pathology(804). The surgical plan is a plan to address the pathology. Forinstance, in the example where the area of interest is the patient'sshoulder, the surgical plan may be selected from an anatomical shoulderarthroplasty, a reverse shoulder arthroplasty, a post-trauma shoulderarthroplasty, or a revision to a previous shoulder arthroplasty. Thesurgical plan may then be tailored to patient (806). For instance,tailoring the surgical plan may involve selecting and/or sizing surgicalitems needed to perform the selected surgical plan. Additionally, thesurgical plan may be tailored to the patient in order to address issuesspecific to the patient, such as the presence of osteophytes. Asdescribed in detail elsewhere in this disclosure, one or more users mayuse mixed reality systems of orthopedic surgical system 100 to tailorthe surgical plan to the patient.

The surgical plan may then be reviewed (808). For instance, a consultingsurgeon may review the surgical plan before the surgical plan isexecuted. As described in detail elsewhere in this disclosure, one ormore users may use mixed reality (MR) systems of orthopedic surgicalsystem 100 to review the surgical plan. In some examples, a surgeon maymodify the surgical plan using an MR system by interacting with a UI anddisplayed elements, e.g., to select a different procedure, change thesizing, shape or positioning of implants, or change the angle, depth oramount of cutting or reaming of the bone surface to accommodate animplant.

Additionally, in the example of FIG. 8, surgical items needed to executethe surgical plan may be requested (810).

As described in the following sections of this disclosure, orthopedicsurgical system 100 may assist various users in performing one or moreof the preoperative steps of FIG. 8.

FIG. 9 illustrates an example welcome page for selecting a surgicalcase, according to an example of this disclosure. The Welcome page,which may be presented by MR visualization device 213 to a user,displays a menu 904 that allows the user to scroll through and select aspecific patient's surgical plan that is stored on and retrieved fromstorage system 206 in system 200 (FIG. 2) or in memory or storage device215 of MR visualization device 213 (FIG. 2).

FIG. 10 illustrates an example of a page of a user interface of a mixedreality system, according to an example of this disclosure, e.g. asproduced for a particular patient's surgical plan selected from thewelcome page of FIG. 9. Using visualization device 213, a user canperceive and interact with UI 522. In the example shown in FIG. 10, UI522 includes a workflow bar 1000 with selectable buttons 1002 thatrepresent a surgical workflow, spanning various surgical procedure stepsfor operations on the humerus and glenoid in a shoulder arthroplastyprocedure. Selection of a button 1002 can lead to display of variousselectable widgets with which the user can interact, such as by usinghand gestures, voice commands, gaze direction, connected lens and/orother control inputs. Selection of widgets can launch various modes ofoperation of MR system 212, display information or images generated byMR system 212, allow the user to further control and/or manipulate theinformation and images, lead to further selectable menus or widgets,etc.

The user can also organize or customize UI 522 by manipulating, movingand orienting any of the displayed widgets according to the user'spreferences, such as by visualization device 213 or other devicedetecting gaze direction, hand gestures and/or voice commands. Further,the location of widgets that are displayed to the user can be fixedrelative to the scene. Thus, as the user's gaze (i.e., eye direction)moves to view other features of the user interface 522, other virtualimages, and/or real objects physically present in the scene (e.g., thepatient, an instrument set, etc.), the widgets may remain stationary anddo not interfere with the user's view of the other features and objects.As yet another example, the user can control the opacity or transparencyof the widgets or any other displayed images or information. The useralso can navigate in any direction between the buttons 1002 on theworkflow bar 1000 and can select any button 1002 at any time during useof MR system 212. Selection and manipulation of widgets, information,images or other displayed features can be implemented based onvisualization device 213 or other device detecting user gaze direction,hand motions, voice commands or any combinations thereof.

In the example of FIG. 10, UI 522 is configured for use in shoulderrepair procedures and includes, as examples, buttons 1002 on workflowbar 1000 that correspond to a “Welcome” page, a “Planning” page, a“Graft” page, a “Humerus Cut” page, an “Install Guide” page, a “GlenoidReaming” page, and a “Glenoid Implant” page. The presentation of the“Install Guide” page may be optional as, in some examples, glenoidreaming may be accomplished using virtual guidance and without theapplication of a glenoid guide.

As shown FIG. 10, the “Planning” page in this example of UI 522 displaysvarious information and images corresponding to the selected surgicalplan, including an image 1006 of a surgical plan file (e.g., a pdf fileor other appropriate media format) that corresponds to the selected plan(including preoperative and postoperative information); a 3D virtualbone model 1008 and a 3D virtual implant model 1010 along with a 3Dimage navigation bar 1012 for manipulating the 3D virtual models 1008,1010 (which may be referred to as 3D images); a viewer 1014 and a viewernavigation bar 1016 for viewing a multi-planar view associated with theselected surgical plan. MR system 212 may present the “Planning” page asa virtual MR object to the user during preoperative phase 302 (FIG. 3).For instance, MR system 212 may present the “Planning” page to the userto help the user classify a pathology, select a surgical plan, tailorthe surgical plan to the patient, revise the surgical plan, and reviewthe surgical plan, as described in steps 802, 804, 806, and 808 of FIG.8.

The surgical plan image 1006 may be a compilation of preoperative (and,optionally, postoperative) patient information and the surgical plan forthe patient that are stored in a database in storage system 206. In someexamples, surgical plan image 1006 can correspond to a multi-pagedocument through which the user can browse. For example, further imagesof pages can display patient information, information regarding theanatomy of interest, postoperative measurements, and various 2D imagesof the anatomy of interest. Yet further page images can include, asexamples, planning information associated with an implant selected forthe patient, such as anatomy measurements and implant size, type anddimensions; planar images of the anatomy of interest; images of a 3Dmodel showing the positioning and orientation of a surgical guideselected for the patient to assist with execution of the surgical plan;etc.

It should be understood that the surgical plan image 1006 can bedisplayed in any suitable format and arrangement and that otherimplementations of the systems and techniques described herein caninclude different information depending upon the needs of theapplication in which the plan image 1006 is used.

Referring again FIG. 10, the Planning page of UI 522 also may provideimages of the 3D virtual bone model 1008 and the 3D model of the implantcomponents 1010 along with navigation bar 1012 for manipulating 3Dvirtual models 1008, 1010. For example, selection or de-selection of theicons on navigation bar 1012 allow the user to selectively viewdifferent portions of 3D virtual bone model 1008 with or without thevarious implant components 1010. For example, the scapula of virtualbone model 1008 and the glenoid implant of implant model 1010 have beende-selected, leaving only the humerus bone and the humeral implantcomponents visible. Other icons can allow the user to zoom in or out,and the user also can rotate and re-orient 3D virtual models 1008, 1010,e.g., using gaze detection, hand gestures and/or voice commands.

The Planning page of UI 522 also provides images of 3D virtual bonemodel 1008 and the 3D model of the implant components 1010 along withnavigation bar 1012 for manipulating 3D virtual models 1008, 1010. Forexample, as shown in FIG. 12, selection or de-selection of icons 1218(FIG. 12) on the navigation bar 1012 allow the user to selectively viewdifferent portions of the 3D virtual bone model 1008 with or without thevarious implant components 1010. In this example, the scapula of virtualbone model 1008 and the glenoid implant of the implant model 1010 havebeen de-selected, leaving only the humerus bone and the humeral implantcomponents visible. Icons 1220 (FIG. 12) allow the user to zoom in orout, and the user also can rotate and re-orient the 3D virtual models1008, 1010 using gaze detection, hand gestures and/or voice commands.

In the example of FIG. 11A, selection of range-of-motion icon 1102 onnavigation bar 1012 of the “Planning” page launches a range-of-motionmode in which the user can test or confirm the selection, placementand/or positioning of the implant components 1010 by simulating variousdifferent motions of the anatomy with the prosthetic implant implantedaccording to the preoperative surgical plan for the patient. In thisexample, by using gaze direction, hand motions detected by motionsensors or other control devices and/or voice commands, the user canselect anatomical motions, such as adduction, abduction,internal/external rotation, elevation, flexion, and extension or anyother joint movement simulation such as movements corresponding to dailyfunctional tasks. In the example of FIG. 11A, a range-of-motion menu1104 includes selectable elements corresponding to different types ofmotions.

In response to receiving an indication of user input to select one ofthe movement (i.e., “Adduction,” “Abduction,” “Internal Rotation 0degrees,” “External Rotation 0 degrees,” “Extension,” and “Flexion”), MRsystem 212 may present an MR animation of 3D virtual model 1008exhibiting the selected movement type. For example, if the user selectedabduction, MR system 212 may present an animation of the humerus of 3Dvirtual model 1008 rotating vertically relative to the scapula of 3Dvirtual model 1008. Furthermore, in the example of FIG. 11A,range-of-motion menu 1104 includes a “Replay Full Motion” element 1125.In response to receiving an indication of user input to select element1125, MR system 212 may present an animation of the humerus of 3Dvirtual model 1008 moving in each of the movement types listed inrange-of-motion menu 1104.

Range-of-motion menu 1104 also lists an impingement angle for each ofthe types of motion. In the example of FIG. 11A, the impingement anglefor a type of motion is an angle at which a bony impingement occurs whenperforming the type of motion. A bony impingement occurs when a bonecontacts another bone or when a moving implanted surgical component(e.g., a cup member connected to the humerus, a talar implant, a tibialimplant, etc.) contacts a bone. Bony impingements may be painful and maycause wear on the bone. Because bony impingements represent contactbetween two hard surfaces, bony impingements also represent thetheoretical limits to ranges of motion. In the example of FIG. 11A, theimpingement angles represent angles determined for a given virtualsurgical plan, including implant components, along with component size,position and angle, specified by the virtual surgical plan.

Visualization of the simulated ranges of motion using MR system 212 canhelp the surgeon confirm the surgical plan or may lead the surgeon toupdate or modify the preoperative surgical plan. For example, in FIG.11A, the user has selected the adduction button on a range-of-motionmenu 1104. A collision 1106 between a scapula and a humerus-mounted cupcomponent during the adduction simulation is highlighted (e.g., in red)in the MR visualization by visualization device 213. In another example,a collision between a talus bone and a tibial implant, or between atalar implant and a tibia, may be highlighted in an MR visualization byvisualization device 213. To further aid the user, MR system 212 mayrotate the 3D models, walk around the 3D models, hide or show parts ofthe 3D models, or perform other actions to observe the 3D models.

If a bony impingement (i.e., a collision) occurs at an angle within thenormal range of motion for a patient, this may indicate to the surgeonthat a change in certain parameters of the surgical plan (e.g., size,type, position or orientation of implant components) may be needed.However, if such a collision occurs at an angle outside the normal rangeof motion for the patient, there may be no need for the surgeon tochange the parameters of the surgical plan. Rather, other tissues of thepatient may stop the motion before a collision occurs. For instance, inthe example of FIG. 11A, the patient's side or tendons attached to thehumerus may prevent collision 1106 from actually occurring. However, thenormal range of motion for abduction is 180° while menu 1104 indicatesthat a bony impingement would occur at an angle of 60°. Thus, with thecurrent parameters of the surgical plan shown in FIG. 11A, the patientwould not even be able to raise their arm to a horizontal position. Thisdisclosure may use the term “premature collision” to refer to a bonyimpingement that occurs within a normal range of motion.

Showing collisions, such as collision 1106, as part of an MRpresentation of animation of a range of motion may help the surgeonunderstand how to change certain parameters of the surgical plan. Forexample, a bony impingement between the humerus and either the acromionor coracoid process may limit the patient's range of motion duringabduction. Thus, in this example, the surgeon may be able to determinethat the ball component should be offset rearward by seeing that apremature bony collision occurs between the patient's humerus andacromion during abduction but no collision occurs between the patient'shumerus and coracoid process during abduction. However, if there is apremature bony collision between the humerus and both the acromion andcoracoid process, the surgeon may determine that a differently sizedball component or wedge is required. Enabling the surgeon to see,rotate, walk around, or otherwise interact with the 3D virtual bonemodel and 3D virtual implant model may help the surgeon make thisdetermination.

FIG. 11B is a flowchart illustrating an example operation to assist insurgical parameter selection, in accordance with a technique of thisdisclosure. Thus, in accordance with a technique of this disclosure, acomputing system, such as MR system 212 or preoperative surgicalplanning system 202 (FIG. 2) may generate, based on medical images of apatient, a 3-dimensional (3D) virtual model of a joint of the patient(1108). The joint may be various types of joints, such as the shoulderjoint, ankle, knee, elbow, or wrist.

Additionally, the computing system may also generate, based on a set ofsurgical parameters, a 3D virtual implant model for the joint (1110). Inan example where the joint is a shoulder joint, the 3D virtual implantmodel may include a ball component, a cup component, and a humeral stemcomponent, as shown in FIG. 11A. Virtual implant models for other jointsor other surgeries on the shoulder joint may include differentcomponents. The set of surgical parameters may indicate sizes, shapes,positions, or other aspects of components of one or more components ofan implant for the joint. Generating the 3D virtual implant model maycomprise selecting and arranging virtual objects that correspond tocomponents of the implant that have the sizes indicated by the set ofsurgical parameters.

Additionally, the computing system may determine a plurality ofimpingement angles based on the 3D virtual bone model and the 3D virtualimplant model (1112). Each respective impingement angle of the pluralityof impingement angles corresponds to a different motion type in aplurality of motion types of the joint. For each respective impingementangle of the plurality of impingement angles, the respective impingementangle indicates an angle at which a bony impingement occurs during themotion type corresponding to the respective impingement angle. In someexamples, the computing system may determine the impingement angles bymoving components of the 3D virtual bone model and 3D virtual implantmodel and detecting where collisions between virtual objects in the 3Dvirtual bone model and 3D virtual implant model occur. In examples wherethe joint is a shoulder joint, the motion types may include adduction,abduction, internal rotation, external rotation, extension, and flexion.In examples where the joint is an ankle joint, the motion types mayinclude plantarflexion and dorsiflexion.

A MR visualization device, such as visualization device 213 (FIG. 2),may present a MR visualization that includes the 3D virtual bone model,the 3D virtual implant model, and visual elements indicating a pluralityof impingement angles (1114). FIG. 11A shows an example of such an MRvisualization. The MR visualization device may present the MRvisualization at various times during a surgical lifecycle. Forinstance, the MR visualization device may present the MR visualizationduring preoperative phase 302 (FIG. 3), intraoperative phase 306, orpostoperative phase 308. Hence, a surgeon or other user may view therange of MR motion visualization when planning a surgery, e.g., as avirtual model, during the course of a surgery, e.g., alone as a virtualmodel or in conjunction with viewing of actual patient anatomy andpresentation of virtual, intra-operative guidance elements, or aftercompletion of a surgery, e.g., as a virtual model.

Furthermore, the MR visualization device may visually indicate in the MRvisualization one or more points at which two or more components of the3D virtual bone model and 3D virtual implant model collide (1116), e.g.,producing an impingement 1106. In some examples, the MR visualizationdevice may indicate the points of collision by presenting, in the MRvisualization, one or more impingement identifiers in areas of the oneor more points. The impingement identifiers may include glowing areas,arrows, highlighted areas, colors, flashing elements, geometric shapes,outlines, or other types of indicators may be used to visually indicatethe points of collision.

In some examples, the MR visualization device may present an animationof the 3D virtual bone model and 3D virtual implant model movingaccording to a motion type of the plurality of motion types. In someexamples, the MR visualization device may present an animation of the 3Dvirtual bone model and 3D virtual implant model moving according to eachmotion type of a plurality of motion types. In either example, the MRvisualization device may present the animation in response to receivingan indication of user input, such as a hand gesture selecting an elementcorresponding to the motion type in menu 1104 or voice command. Forinstance, with respect to FIG. 11A, the MR visualization device maypresent an animation of the humerus moving according to anabduction/adduction motion type relative to the scapula in response toreceiving an indication of user input to select the adduction element ofmenu 1104, the abduction element of menu 1104, or the “Replay FullMotion” element of menu 1104. In examples where the MR visualizationdevice presents the animation, the MR visualization device may visuallyindicate, during the animation, one or more points at which two or morecomponents of the 3D virtual bone model and 3D virtual implant modelcollide. Furthermore, in some examples, the MR visualization device maygenerate an audible or tactile notification during the animation whenthe two or more components of the 3D virtual bone model and 3D virtualimplant model collide. For instance, a speaker of MR visualizationdevice may output the audible notification (e.g., as a beeping orclicking sound) when the animation shows a frame in which a collisionoccurs. A vibration unit of MR visualization device may generate tactilenotification is a vibration when the animation shows a frame in which acollision occurs.

As discussed elsewhere in this disclosure, a user may use theinformation in the MR visualization regarding the impingement angles andcollision points to determine whether to make adjustments to thesurgical parameters.

Returning to the example of FIG. 10, the Planning page presented byvisualization device 213 also includes multi-planar image viewer 1014(e.g., a DICOM viewer) and navigation bar 1016 that allow the user toview patient image data and to switch between displayed slices andorientations. For example, the user can select 2D Planes icons 1026 onnavigation bar 1016 so that the user can view the 2D sagittal andcoronal planes of the patient's body in multi-planar image viewer 1014.

Workflow bar 1000 in FIG. 10 includes further pages that correspond tosteps in the surgical workflow for a particular orthopedic procedure(here, a shoulder repair procedure). In the example of FIG. 10, workflowbar 1000 includes elements labeled “Graft,” “Humerus Cut,” “InstallGuide,” “Glenoid Reaming,” and “Glenoid Implant” that correspond toworkflow pages for steps in the surgical workflow for a shoulder repairprocedure. In general, these workflow pages include information that canbe useful for a health care professional during planning of or duringperformance of the surgical procedure, and the information presentedupon selection of these pages is selected and organized in a manner thatis intended to minimize disturbances or distractions to the surgeonduring a procedure. Thus, the amount of displayed information isoptimized and the utility of the displayed information is maximized.These workflow pages may be used as part of intraoperative phase 306(FIG. 3) to guide a surgeon, nurse or other medical technician throughthe steps in a surgical procedure. In some examples, these workflowpages may be used as part of preoperative phase 302 (FIG. 3) to enable auser to visualize 3-dimensional models of objects involved in varioussteps of a surgical workflow.

In the example shown, each workflow page that can be selected by theuser (e.g., a surgeon) can include an Augment Surgery widget, such asAugment Surgery widget 1300 (shown in FIG. 13), that, when selected,launches an operational mode of MR system 212 in which a user using(e.g., wearing) visualization device 213 (FIG. 2) can see the details(e.g., virtual images of details) of the surgical plan projected andmatched onto the patient bone and use the plan intraoperatively toassist with the surgical procedure. In general, the Augment Surgery modeallows the surgeon to register the virtual 3D model of the patient'sanatomy of interest (e.g., glenoid) with the observed real anatomy sothat the surgeon can use the virtual surgical planning to assist withimplementation of the real surgical procedure, as will be explained infurther detail below. There may be different Augment Surgery widgets foreach of the steps of the surgery that the surgeon uses during actualsurgery. The Augment Surgery widgets for different steps may includedifferent text, control, icons, graphics, etc.

In this example of a shoulder repair procedure, and with reference FIG.10, the workflow pages of UI 522 that can be used by the surgeon include“Graft”, “Humerus Cut”, “Install Guide”, “Glenoid Reaming”, and “GlenoidImplant”. The “Graft” step and “Install Guide” steps may be optional.For example, it may not be necessary to take a graft in every procedureand the use of a glenoid reaming guide may not be necessary if MRreaming axis guidance is presented to the user by visualization device213. A user may view the workflow pages during the preoperative phase302, during the intraoperative phase 306, or at other times. It may behelpful to a surgeon to view the workflow pages during the preoperativephase 302 in order to tailor a surgical plan for the patient, to reviewthe steps of a surgical plan, or perform other tasks. It may be helpfulto a surgeon to view the workflow pages in the intraoperative phase 306to refresh the surgeon on the anatomy of the patient involved in thecorresponding surgical steps, to obtain information on how to performcertain actions during the corresponding surgical steps, to takeinventory of surgical instruments, implants or other surgical itemsneeded in the surgical steps, and so on. As mentioned, each of theworkflow pages generally corresponds to a step in the workflow for theparticular surgical procedure. Thus, for example, the Graft page allowsthe user to visualize a bone graft 1402 (FIG. 14) matched for aparticular patient and provides the user with sufficient information forselecting, designing and/or modifying the shape and dimensions of bonegraft 1402, if desired. As an example, bone graft 1402 may be a bonegraft taken from the humerus or another bone.

As another example, with reference to FIG. 15A, the Humerus Cut pagepresents the user with a 3D model 1504 of the humerus and a 3D model ofthe humeral implant components 1506, and a cutting plane 1508 on thehumeral head, e.g., for application of a cutting tool such as a rotaryor reciprocating saw to remove a portion of the humeral head. Inresponse to selection of other items on menu 1510 on this page,visualization device 213 can remove the 3D model of the humeral implantcomponents 1506 from the presented imagery and provide images of thehumeral head before and after cutting, as examples.

FIGS. 15B-15D are examples of hiding virtual objects in humerus cut pageof FIG. 15A, according to an example of this disclosure. As noted above,menu 1510 of FIG. 15A enables a user to selectively add or remove 3Dmodels of components of the patient's humerus and implant components1506. For instance, in the examples of FIGS. 15B-15D, the user hasremoved the 3D models of the implant components 1506. In FIG. 15B, theuser has chosen to view the humerus and humeral head. In FIG. 15C, theuser has chosen to view the humerus and not the humeral head. In FIG.15D, the user has chosen to view the humeral head and not the humerus.In this example, the presentation of the humerus without the humeralhead may be selected to show a humeral cutting plane for removal of theportion of the humeral head according to the virtual surgical plan,e.g., to permit placement of an implant component such as a humeral ballor plate component on the humerus in the course of the surgicalprocedure.

As another example (not shown), an Osteophytes item can be presented forselection on the menu page by visualization device 213. When selected,an osteophytes assessment feature is activated in which osteophytes canbe identified, projected, highlighted or hidden on the patient bone inthe MR visualization presented to the user by visualization device 213.

With reference to FIG. 16, the Install Guide page allows the user tovisualize a physical position of a patient-specific or patient-matchedguide 1600, e.g., for guidance of a drill to place a reaming guide pinin the glenoid bone, on the patient's glenoid 1602 in order to assistwith the efficient and correct placement of the guide 1600 during theactual surgical procedure. Selection of items on menu 1604 can removefeatures from the 3D images or add other parameters of the surgicalplan, such as a reaming axis 1606, e.g., by voice commands, gazedirection and/or hand gesture selection. Placement of guide 1600 may beunnecessary for procedures in which visualization device 213 presents avirtual reaming axis or other virtual guidance, instead of a physicalguide, to guide a drill for placement of a reaming guide pin in theglenoid bone. The virtual guidance or other virtual objects presented byvisualization device 213 may include, for example, one or more 3Dvirtual objects. In some examples, the virtual guidance may include 2Dvirtual objects. In some examples, the virtual guidance may include acombination of 3D and 2D virtual objects.

With reference to FIG. 17, the Glenoid Implant page allows the user tovisualize the orientation and placement of a glenoid implant 1700 andbone graft 1402 on glenoid 1602.

It should be understood that the workflow pages illustrated anddescribed herein are examples and that UI 522 can include fewer, more,or different pages. For example, in applications of MR system 212 forprocedures involving other patient anatomies, such as the ankle, foot,knee, hip or elbow, UI 522 can include pages corresponding to theparticular steps specific to the surgical workflow for those procedures.

The images displayed on UI 522 of MR system 212 can be viewed outside orwithin the surgical operating environment and, in spectator mode, can beviewed by multiple users outside and within the operating environment atthe same time. In some circumstances, such as in the operatingenvironment, the surgeon may find it useful to use a control device 534to direct visualization device 213 such that certain information shouldbe locked into position on a wall or other surface of the operatingroom, as an example, so that the information does not impede thesurgeon's view during the procedure. For example, relevant surgicalsteps of the surgical plan can be selectively displayed and used by thesurgeon or other care providers to guide the surgical procedure.

In various some examples, the display of surgical steps can beautomatically controlled so that only the relevant steps are displayedat the appropriate times during the surgical procedure.

As discussed above, surgical lifecycle 300 may include an intraoperativephase 306 during which a surgical operation is performed. One or moreusers may use orthopedic surgical system 100 in intraoperative phase306.

FIG. 18 is a conceptual diagram illustrating an example setting in whicha set of one or more users use MR systems of orthopedic surgical system100 during intraoperative phase 306. In the example of FIG. 18, asurgeon may wear a visualization device (e.g., visualization device 213)of a first MR system 1800A (e.g., MR system 212). The visualizationdevice of MR system 1800A may present MR intraoperative guidance content1802 to the surgeon during intraoperative phase 306. As described indetail elsewhere in this disclosure, MR intraoperative guidance content1802 may help the surgeon perform for a surgical operation.

Additionally, in the example of FIG. 18, one or more other users may usevisualization devices of MR systems of orthopedic surgical system 100 toview MR intraoperative guidance content 1802. For example, a nurse mayuse a visualization device of an MR system 1800B of orthopedic surgicalsystem 100. Furthermore, in the example of FIG. 18, a technician may usea visualization device of an MR system 1800C of orthopedic surgicalsystem 100. In the example of FIG. 18, a second surgeon may use avisualization device of an MR system 1800D of orthopedic surgical system100. MR systems 1800A, 1800B, 1800C, and 1800D may be referred to hereincollectively as “MR systems 1800.” In some examples, a television orother display device may present the view of the surgeon, which mayinclude virtual objects, to one or more other individuals, such as anurse, surgeon, or technician.

Two or more of the individuals described above (e.g., the first surgeon,the nurse, the technician, the second surgeon) may view the same ordifferent MR intraoperative guidance content 1802 at the same time. Inexamples where two or more of the individuals are viewing the same MRintraoperative guidance content 1802 at the same time, the two or moreindividuals may concurrently view the same MR intraoperative guidancecontent 1802 from the same or different perspectives.

One or more users may use orthopedic surgical system 100 in anintraoperative setting. For example, the users may manipulate a userinterface presented by MR systems 1800 so that the users can view avirtual surgical plan intraoperatively. For instance, in this example,the users may view a 3D virtual model of an anatomy of interest (e.g., a3-dimensional virtual bone model of an anatomy of interest).

In some examples, one or more users, including at least one surgeon, mayuse orthopedic surgical system 100 in an intraoperative setting toperform shoulder surgery. FIG. 19 is a flowchart illustrating examplestages of a shoulder joint repair surgery. As discussed above, FIG. 19describes an example surgical process for a shoulder surgery. Thesurgeon may wear or otherwise use visualization device 213 during eachstep of the surgical process of FIG. 10. In other examples, a shouldersurgery may include more, fewer, or different steps. For example, ashoulder surgery may include step for adding a bone graft, addingcement, and/or other steps. In some examples, visualization device 213may present virtual guidance to guide the surgeon, nurse, or otherusers, through the steps in the surgical workflow.

In the example of FIG. 19, a surgeon performs an incision process(1900). During the incision process, the surgeon makes a series ofincisions to expose a patient's shoulder joint. In some examples, an MRsystem (e.g., MR system 212, MR system 1800A, etc.) may help the surgeonperform the incision process, e.g., by displaying virtual guidanceimagery illustrating how to where to make the incision.

Furthermore, in the example of FIG. 19, the surgeon may perform ahumerus cut process (1902). During the humerus cut process, the surgeonmay remove a portion of the humeral head of the patient's humerus.Removing the portion of the humeral head may allow the surgeon to accessthe patient's glenoid. Additionally, removing the portion of the humeralhead may allow the surgeon to subsequently replace the portion of thehumeral head with a humeral implant compatible with a glenoid implantthat the surgeon plans to implant in the patient's glenoid.

As discussed above, the humerus preparation process may enable thesurgeon to access the patient's glenoid. In the example of FIG. 19,after performing the humerus preparation process, the surgeon mayperform a registration process that registers a virtual glenoid objectwith the patient's actual glenoid bone (1904) in the field of viewpresented to the surgeon by visualization device 213.

FIG. 20A illustrates an example of a technique 2000 for registering a 3Dvirtual bone model 1008 with a real observed bone structure 2200 of apatient. In other words, FIG. 20A is an example of a process flow, e.g.,performed by visualization device 213, for registering a virtual bonemodel with an observed bone that is implemented in a mixed realitysystem, such as the mixed reality system 212 of FIG. 2. FIG. 20B,described below, illustrates another technique 2018 for registering 3Dvirtual bone model 1008 bone structure 2200, using physical registrationmarkers.

With further reference to FIG. 20A, 3D virtual bone model 1008 may be amodel of all or part of one or more bones. The process flow of FIG. 20Amay be performed as part of the registration process of step 1904 ofFIG. 19. The registration process may be carried out in two steps:initialization and optimization (e.g., minimization). Duringinitialization, the user of MR system 212 uses the visualization device213 in conjunction with information derived from the preoperativevirtual planning system 102, the orientation of the user's head (whichprovides an indication of the direction of the user's eyes (referred toas “gaze” or “gaze line”), rotation of the user's head in multipledirections, sensor data collected by the sensors 530, 532 and/or 533 (orother acquisitions sensors), and/or voice commands and/or hand gesturesto visually achieve an approximate alignment of the 3D virtual bonemodel 1008 with observed bone structure 2200. More particularly, atblock 2002, a point or region of interest on the surface of the virtualbone model 1008 and a virtual normal vector to the point (or region) ofinterest on the surface of the region are identified during thepreoperative planning using the virtual planning system 102.

At block 2004, MR system 212 connects the identified point (or region)of interest to the user's gaze point (e.g., a central point in the fieldof view of visualization device 213). Thus, when the head of the user ofvisualization device 213 is then moved or rotated, the virtual bonemodel 1008 also moves and rotates in space.

In the example of a shoulder arthroplasty procedure, the point ofinterest on the surface of virtual bone model 1008 can be an approximatecenter of the virtual glenoid that can be determined by using a virtualplanning system 102, such as the BLUEPRINT™ planning system. In someexamples, the approximate center of the virtual glenoid can bedetermined using a barycenter find algorithm, with the assistance ofmachine learning algorithms or artificial intelligence systems, or usinganother type of algorithm. For other types of bone repair/replacementprocedures, other points or regions of the bone can be identified andthen connected to the user's gaze line or gaze point.

The ability to move and rotate virtual bone model 1008 in space aboutthe user's gaze point alone generally is not sufficient to orientvirtual bone model 1008 with the observed bone. Thus, as part of theinitialization procedure, MR system 212 also determines the distancebetween visualization device 213 and a point (or points) on the surfaceof the observed bone in the field of view of visualization device 213and the orientation of that surface using sensor data collected from thedepth, optical, and motion sensors 530, 532, 533 (block 2008). Forexample, a glenoid is a relatively simple surface because, locally, itcan be approximated by a plane. Thus, the orientation of the glenoidsurface can be approximated by determining a vector that is normal(i.e., perpendicular) to a point (e.g., a central point) on the surface.This normal vector is referred to herein as the “observed normalvector.” It should be understood, however, that other bones may havemore complex surfaces, such as the humerus or knee. For these morecomplex cases, other surface descriptors may be used to determineorientation.

Regardless of the particular bone, distance information can be derivedby MR system 212 from depth camera(s) 532. This distance information canbe used to derive the geometric shape of the surface of an observedbone. That is, because depth camera(s) 532 provide distance datacorresponding to any point in a field of view of depth camera(s) 532,the distance to the user's gaze point on the observed bone can bedetermined. With this information, the user can then move 3D virtualbone model 1008 in space and approximately align it with the observedbone at a point or region of interest using the gaze point (block 2010in FIG. 20A). That is, when the user shifts gaze to observed bonestructure 2200 (block 2006 in FIG. 20A), virtual bone model 1008 (whichis connected to the user's gaze line) moves with the user's gaze. Theuser can then align 3D virtual bone model 1008 with observed bonestructure 2200 by moving the user's head (and thus the gaze line), usinghand gestures, using voice commands, and/or using a virtual interface toadjust the position of virtual bone model 1008. For instance, once 3Dvirtual bone model 1008 is approximately aligned with observed bonestructure 2200, the user may provide a voice command (e.g., “set”) thatcauses MR system 212 to capture the initial alignment. The orientation(“yaw” and “pitch”) of the 3D model can be adjusted by rotating theuser's head, using hand gestures, using voice commands, and/or using avirtual interface which rotate 3D virtual bone model 1008 about theuser's gaze line so that an initial (or approximate) alignment of thevirtual and observed objects can be achieved (block 2012 in FIG. 20A).In this manner, virtual bone model 1008 is oriented with the observedbone by aligning the virtual and observed normal vectors. Additionaladjustments of the initial alignment can be performed as needed. Forinstance, after providing the voice command, the user may provideadditional user input to adjust an orientation or a position of virtualbone model 1008 relative to observed bone structure 2200. This initialalignment process is performed intraoperatively (or in real time) sothat the surgeon can approximately align the virtual and observed bones.In some examples, such as where the surgeon determines that the initialalignment is inadequate, the surgeon may provide user input (e.g., avoice command, such as “reset”) that causes MR system 212 to release theinitial alignment such that point 2106 is again locked to the user'sgaze line.

At block 2014 of FIG. 20A, when the user detects (e.g., sees) that aninitial alignment of 3D virtual bone model 1008 with observed bonestructure 2200 has been achieved (at least approximately), the user canprovide an audible or other perceptible indication to inform MR system212 that a fine registration process (i.e., execution of an optimization(e.g., minimization) algorithm) can be started. For instance, the usermay provide a voice command (e.g., “match”) that causes MR system 212 toexecute a minimization algorithm to perform the fine registrationprocess. The optimization process can employ any suitable optimizationalgorithm (e.g., a minimization algorithm such as an Iterative ClosestPoint or genetic algorithm) to perfect alignment of virtual bone model1008 with observed bone structure 2200. At block 2016 of FIG. 20A, uponcompletion of execution of the optimization algorithm, the registrationprocedure is complete.

FIG. 21 is a conceptual diagram illustrating steps of an exampleregistration process for a shoulder arthroplasty procedure. FIG. 22 is aconceptual diagram illustrating additional steps of the exampleregistration process of the shoulder arthroplasty procedure of FIG. 21.In FIG. 21, a gaze line 2104 of a user of visualization device 213 isconnected with the previously identified point of interest (or gazepoint) 2106 on a surface 2108 of 3D virtual bone model 1008 (a glenoid).FIG. 21 also shows a virtual normal vector (Nv) 2110 to point 2106 onsurface 2108. In FIG. 22, the user of visualization device 213 shiftsgaze line 2104 to a region of interest 2202 on surface 2204 of observedbone structure 2200. Because gaze line 2104 is connected to the centerpoint 2106 of virtual bone model 1008, shifting gaze line 2104 alignsvirtual center point 2106 of virtual bone model 1008 with the observedregion of interest 2202. However, as shown in FIG. 22, simply shiftingthe gaze aligns the center points/regions 2106, 2202, but may notproperly orient the virtual bone model 1008 (shown in dashed lines) withobserved bone structure 2200. Once an observed normal vector (NO) 2206is determined as discussed above, visualization device 213 can adjustthe orientation (pitch and yaw) of virtual bone model 1008 until theproper orientation is achieved (shown in dotted lines) and virtualnormal vector (VN) 2110 is aligned with observed normal vector 2206. Theuser may rotate virtual bone model 1008 around the aligned axes passingthrough the glenoid for proper alignment of virtual bone model 1008 withthe corresponding real bone.

FIG. 23 and FIG. 24 are conceptual diagrams illustrating an exampleregistration process for a shoulder arthroplasty procedure. Similar tothe registration process shown in FIG. 21, FIG. 23 illustrates theviewpoint of a user of visualization device 213. As shown in FIG. 23,point of interest 2106 is shown on virtual bone model 1008. As discussedabove, as the gaze of the user is connected to point 2106, the user maymove virtual bone model 1008 by shifting their gaze, in which casevisualization device 213 detects the gaze shift and moves the virtualbone model in a corresponding manner. As shown in FIG. 24, to alignvirtual bone model 1008 with observed bone structure 2200, the user mayshift their gaze in the direction indicated by arrow 2400.

For some surgical bone repair procedures, such as shoulderarthroplasties, alignment and orientation of the virtual and observedbone using only the user's gaze can be challenging. These challengesarise due to many factors, including that the bone (e.g., glenoid) islocated quite deep under the skin so that even after the surgicalincision is made, it can be difficult to position the visualizationdevice 213 close to the bone; shadows may obscure the bone; the entirebone surface of interest may not be visible; and it can be difficult forthe user to maintain a steady and stable gaze which can result ininstability in the positioning of the virtual bone. In some examples, toaddress these challenges, the registration procedure can be facilitatedthrough the use of virtual landmark(s) placed at specific location(s) onthe bone (e.g., the center of the glenoid for a shoulder arthroplastyprocedure). In such examples, the location at which the virtual landmarkis placed and the surface normal at that location can be used toautomatically determine the initialization transformation (orregistration transformation) for the virtual and observed bones. Ifdesired, the alignment achieved between the virtual and observed boneusing the virtual landmark can be further adjusted by the user usingvoice commands, hand gestures, virtual interface buttons, and/or bypositioning additional virtual markers at various locations on the bonesurface.

FIG. 25 illustrates an example registration procedure using a virtualmarker 2500. FIG. 26 is a conceptual diagram illustrating additionalsteps of the example registration procedure of FIG. 20A using a virtualmarker. In the example of FIG. 25 and FIG. 26, the user of visualizationdevice 213 shifts a gaze line 2104 to set virtual marker 2500 at acenter region 2202 (e.g., center point) of observed bone structure 2200.With the help of the virtually positioned marker 2500, the virtualnormal vector 2110 and the observed normal vector 2206, theinitialization transformation between virtual bone model 1008 andobserved bone structure 2200 can be determined. Then, the optimizationalgorithm (or registration algorithm) is executed, as described above,in order to obtain an optimal registration between virtual bone model1008 and observed bone structure 2200.

In some examples, the initialization procedure can be implemented basedon a region of interest on the bone surface instead of a point ofinterest. In such examples, the image data collected by the depth and/oroptical camera(s) 530, 532 (FIG. 5) of visualization device 213 can beprocessed to detect surface descriptors that will facilitateidentification of the position and orientation of the observed bone andto determine an initialization transformation between the virtual andobserved bones.

As discussed above, in some examples, the initialization may be aided bythe user (e.g., aided by the user shifting gaze line 2104 to set virtualmarker 2500 at a center region 2202 of observed bone structure 2200). Insome examples, MR system 212 may perform the entire registration process(e.g., including any initialization steps) with minimal or no aid fromthe user. For instance, MR system 212 may process the image datacollected by the depth and/or optical camera(s) 530, 532 (FIG. 5) toautomatically identify a location of the anatomy of interest (e.g.,observed bone structure 2200). As such, MR system 212 may register avirtual model of a portion of anatomy to a corresponding observedportion of anatomy in response to the user looking at the portion ofanatomy (e.g., the surgeon, while wearing visualization device 213, maymerely look at the portion of anatomy). MR system 212 may automaticallyidentify the location using any suitable technique. For example, MRsystem 212 may use a machine learned model (i.e., use machine learning,such as a random forest algorithm) to process the image data andidentify the location of the anatomy of interest.

In more general terms, the registration methods described with referenceto FIG. 20A and FIG. 20B can be viewed as determining a first localreference coordinate system with respect to the 3D virtual model anddetermining a second local reference coordinate system with respect tothe observed real anatomy. In some examples, MR system 212 also can usethe optical image data collected from optical cameras 530 and/or depthcameras 532 and/or motion sensors 533 (or any other acquisition sensor)to determine a global reference coordinate system with respect to theenvironment (e.g., operating room) in which the user is located. Inother examples, the global reference coordinate system can be defined inother manners. In some examples, depth cameras 532 are externallycoupled to visualization device 213, which may be a mixed realityheadset, such as the Microsoft HOLOLENS™ headset or a similar MRvisualization device. For instance, depth cameras 532 may be removablefrom visualization device 213. In some examples, depth cameras 532 arepart of visualization device 213, which again may be a mixed realityheadset. For instance, depth cameras 532 may be contained within anouter housing of visualization device 213.

The registration process may result in generation of a transformationmatrix that then allows for translation along the x, y, and z axes ofthe 3D virtual bone model and rotation about the x, y and z axes inorder to achieve and maintain alignment between the virtual and observedbones.

In some examples, one or more of the virtual markers can be replacedand/or supplemented with one or more physical markers, such as opticalmarkers or electromagnetic markers, as examples. FIGS. 30A-30Eillustrate examples of physical markers positioned around the realobserved bone structure 2200. In general, the one or more physicalmarkers may be positioned at various positions on or around the objectbeing registered (e.g., real observed bone structure 2200 or a tool). Asshown in the examples of FIG. 30A and FIG. 30B, a fixed optical marker3010 may be used in a shoulder arthroplasty procedure to define thelocation of the acromion of the scapula on the real observed bonestructure 2200. In the example of FIG. 30A, fixed optical marker 3010may include a planar fiducial marker 3010A on a single face of theoptical marker. In the example of FIG. 30B, fixed optical marker 3010may include planar fiducial markers 3010A on multiple faces of theoptical marker. Where a physical marker includes fiducial markers ofmultiple faces, the fiducial markers may be the same on every face ordifferent faces may include different fiducial markers. As shown inFIGS. 30A and 30B, the fiducial marker may be positioned on a portion ofthe physical marker that is proximal to a tip 3010B of the marker 3010.In some examples, MR system 212 may obtain a distance between a featureof the fiducial marker (e.g., a centroid or center point) and the tip ofthe physical marker. As one example, the distance may be predeterminedand stored in a memory of MR system 212. As another example, MR system212 may determine the distance based on optical characteristic of thefiducial marker (i.e., the distance may be encoded in the fiducialmarker).

As shown in the example of FIG. 30C, physical markers 3012A-3012D(collectively, “physical markers 3012”) can be positioned at variouspositions around the real observed bone structure 2200. In someexamples, the various positions of the physical markers may correspondto positions (e.g., attachment points) of patient matched guide 1600 ofFIG. 16. Knowledge of the acromion location and the location of thecenter of the glenoid (which may be set virtually) may allow MR system212 to automatically initialize/register the virtual bone without theneed for the user to employ head movements and rotations.

In general, the physical markers may be placed anywhere. For instance,the physical markers can be attached to the patient (e.g., non-sterilefield), surgically exposed anatomy (sterile field), instruments,anywhere in surgical field of view, or any other suitable location.

The physical markers can be any type of marker that enablesidentification of a particular location relative to the real observedobject (e.g., bone structure 2200). Examples of physical markersinclude, but are not necessarily limited to, passive physical markersand active physical markers. Passive physical markers may have physicalparameters that aid in their identification by MR system 212. Forinstance, physical markers may have a certain shape (e.g., sphericalmarkers that may be attached to the real observed bone structure 2200),and/or optical characteristics (e.g., reflective materials, colors(e.g., colors, such a green, that are more visible in a surgicalenvironment), bar codes (including one-dimensional or two-dimensionalbars, such as QR codes), or the like) that aid in their identificationby MR system 212. The passive physical markers can be three-dimensionalor two-dimensional. Passive physical markers may be considered passivein that their presence/position is passively detected by MR system 212.The passive physical markers may be flat or flexible two-dimensionalstickers having planar fiducial markers that can be adhesively mountedto bone, tools or other structures, e.g., via an adhesive back layerexposed upon removal of a release layer. Alternatively, passive physicalmarkers may be fixed to bone, e.g., with surgical adhesive, screws,nails, clamps and/or other fixation mechanisms.

As shown in the example of FIG. 30D, stickers 3016A-3016C that includeplanar fiducial markers are shown as being attached to, or around,observed ankle 3014. Additionally, sticker 3018 that includes a planarfiducial marker is shown as being attached to drill 3020.

Active physical markers may perform one or more actions that aid intheir identification by MR system 212. For instance, active physicalmarkers may output signals (e.g., electromagnetic signals) that aid intheir identification by MR system 212. Examples of active physicalmarkers include, but are not limited to, sensors or transmitters for thetrakSTAR™ and/or driveBAY™ systems available from Northern Digital Inc.

Electromagnetic tracking (i.e., tracking using electromagnetic physicalmarkers, referred to as “EM tracking”) may be accomplished bypositioning sensors within a magnetic field of known geometry, which maybe created by a field generator (FG). The sensors may measure magneticflux or magnetic fields. A tracking device may control the FG andreceive measurements from the sensors. Based on the receivedmeasurements, the tracking device may determine the locations/positionsof the sensors. A more detailed description on EM tracking may be foundin Alfred M. Franz et. al, “Electromagnetic Tracking in Medicine—AReview of Technology, Validation, and Applications,” IEEE TRANSACTIONSON MEDICAL IMAGING, VOL. 33, NO. 8, AUGUST 2014.

FIG. 30E illustrates an example of electromagnetic physical markerspositioned around the real observed bone structure 2200. As illustratedin FIG. 30E, electromagnetic (EM) tracking system 3022 (which may beincluded within MR system 204 of FIG. 2) includes electromagnetic (EM)tracker 3024, field generator 3004, and one or more EM physical markers3028A and 3028B (collectively “EM physical markers 3028”). EM physicalmarkers 3006 may be positioned near and/or attached to the object to betracked (e.g., observed bone structure 2200, an instrument, or the like)using the techniques described above. For instance, EM physical markers3006 may be attached using surgical adhesive, screws, nails, clampsand/or other fixation mechanisms.

Field generator 3004 may be configured to output/generate a magneticfield with a known geometry. Examples of field generator 3004 include,but are not necessarily limited to, permanent magnets or byelectromagnetism. In the electromagnetism case, the structure of themagnetic reference field may be governed by the law of Biot-Savart. Thegeometry of the emitting coil assembly and the type of current sentthrough the coils determines the shape and the geometric properties ofthe aforementioned field.

EM tracker 3024 may be configured to control the operation of fieldgenerator 3004. For instance, EM tracker 3024 may control parameters ofthe EM field generated by field generator 3004 by adjusting the currentflowing through coils of field generator 3004. EM tracker 3024 mayreceive signals from EM physical markers 3006. For instance, EM trackermay receive measurements of magnetic flux and/or magnetic fields from EMphysical markers 3006. EM tracker 3024 may determine thepositions/orientations/locations of EM physical markers 3006 based onthe received measurements.

EM tracker 3024 and field generator 3004 may each be standalonecomponents, may be integrated into a single component, or may beincluded within another component. For instance, EM tracker 3024 may beincluded within MR system 212 of FIG. 2. EM tracker 3024 may output thedetermined positions/orientations/locations of EM physical markers 3006to one or more other components of MR system 204 (e.g., processingdevices 210).

EM tracking system 3022 may be utilized to perform registration with orwithout the use of other markers. As one example, EM tracking system3022 may be utilized to register a virtual model of a bone to acorresponding observed bone without the use of other physical markers orvirtual markers. As another example, EM tracking system 3022 may beutilized to register a virtual model of a bone to a correspondingobserved bone in conjunction with other physical markers and/or virtualmarkers.

FIG. 20B illustrates an example of a technique 2018 for registering a 3Dvirtual bone model 1008 with a real observed bone structure 2200 of apatient using physical markers (e.g., any combination of passive andactive physical markers). In other words, FIG. 20B is an example of aprocess flow, e.g., performed by visualization device 213, forregistering a virtual bone model with an observed bone that isimplemented in a mixed reality system, such as the mixed reality system212 of FIG. 2. 3D virtual bone model 1008 may be a model of all or partof one or more bones. The process flow of FIG. 20B may be performed aspart of the registration process of step 1904 of FIG. 19. As describedbelow, the registration process of FIG. 20B may be used in addition to,or in place of, the registration process of FIG. 20A.

In operation, the practitioner may place one or more physical markers atspecific positions. In some examples, MR system 212 may outputinstructions as to where the practitioner should place the physicalmarkers. The prescribed locations may correspond to specific locationson a virtual model that corresponds to the observed bone structure 2200.For instance, in one example, visualization device 213 may displayinstructions for the practitioner to attach the physical markers (e.g.,with surgical adhesive, screws, nails, clamps and/or other fixationmechanisms) at locations corresponding to positions of patient matchedguide 1600 of FIG. 16 (e.g., regardless of whether patient matched guide1600 is available for use). In other words, the practitioner may attachthe physical makers at the locations where the patient matched guide1600 would attach, even if patient matched guide 1600 is not present. Inother examples, the prescribed locations may be indicated by text,graphical or audible information to cause the surgeon to selectcorresponding locations on the physical bone or tool(s) for attachmentor other placement of the markers. For instance, MR system 212 mayoutput graphic information to guide the surgeon in attaching tip 3010Bof optical marker 3010 of FIG. 30A to the acromion of the scapula.

MR system 212 may utilize data from one or more sensors (e.g., one ormore of sensors 614 of visualization device 213 of FIG. 6) to identifythe location of the physical markers (2020). For instance, MR system 212may use data generated by any combination of depth sensors 532 and/oroptical sensors 530 to identify a specific position (e.g., coordinates)of each of the physical markers. As one specific example, MR system 212may utilize optical data generated by optical sensors 530 to identify acentroid of optical marker 3010A of FIG. 30A. MR system 212 may thenutilize depth data generated by depth sensors 532 and/or optical datagenerated by optical sensors 530 to determine a position and/ororientation of the identified centroid. MR system 212 may determine adistance between the centroid and an attachment point of the physicalmarker. For instance, MR system 212 may determine a distance between acentroid of fiducial marker 3010A and tip 3010B of optical marker 3010of FIG. 30A. Based on the determined distance (i.e., between thecentroid and the attachment point) and the determinedposition/orientation of the centroid, MR system 212 may determine aposition/orientation of the attachment point.

MR system 212 may register the virtual model with the observed anatomybased on the identified positions (2022) of the physical markers. Forinstance, where the physical markers are placed on the observed bonestructure 2200 at locations that correspond to specific location(s) onthe virtual model that corresponds to the observed bone structure 2200,MR system 212 may generate a transformation matrix between the virtualmodel and the observed bone structure 212. This transformation matrixmay be similar to the transformation matrix discussed above in that itallows for translation along the x, y, and z axes of the virtual modeland rotation about the x, y and z axes in order to achieve and maintainalignment between the virtual and observed bones. In some examples,after registration is complete, MR system 212 utilize the results of theregistration to perform simultaneous localization and mapping (SLAM) tomaintain alignment of the virtual model to the corresponding observedobject.

As discussed in further detail below, MR system 212 may display, basedon the registration, virtual guidance for preparing the observed anatomyfor attachment of a prosthetic or virtual guidance for attaching theprosthetic to the observed anatomy (2024). For instance, MR system 212may provide virtual guidance as described below with reference to anycombination of FIGS. 34-71.

As discussed above, the physical markers may be used in addition to, orin place of, the virtual markers (e.g., virtual marker 2500). In otherwords, MR system 212 may perform registration of a virtual model of abone to corresponding observed bone using any combination of physicaland virtual markers. In some examples, using physical markers (eitheralone or with virtual markers) may enable MR system 212 to reduce theamount of time required to perform registration and/or may result inmore accurate registration.

In some examples, MR system 212 may use one of virtual markers orphysical markers as a primary registration marker and use the other as asecondary, or supplemental, registration marker. As one example, MRsystem 212 may begin a registration process by attempting to performregistration using the primary registration marker. In such examples, ifMR system 212 is not able to adequately complete registration (e.g.,cannot generate a mapping, such as a transformation matrix, between thevirtual and observed anatomy) using only the primary registrationmarker, MR system 212 may attempt to perform registration using only thesecondary registration marker or a combination of the primaryregistration marker and the secondary registration marker. In onespecific example, if MR system 212 is not able to adequately completeregistration using only virtual marker(s), MR system 212 may attempt toperform registration using only physical marker(s) or a combination ofvirtual registration marker(s) and physical registration marker(s).

In situations where MR system 212 is not able to adequately completeregistration using only the primary registration marker, MR system 212may output a request for the practitioner to perform one or more actionsto enable registration using the secondary registration marker. As oneexample, where the secondary registration marker is a physical marker,MR system 212 may output a request for the practitioner to position aphysical marker at a particular location relative to the observedanatomy. As another example, where the secondary registration marker isa virtual marker, MR system 212 may output a request and correspondinggraphical user interface (e.g., 3D virtual bone model 1008) for thepractitioner to perform the initial alignment procedure described abovewith reference to FIG. 20A.

In some examples, the practitioner may remove the physical markers(e.g., after registration is complete). For instance, after MR system212 has completed the registration process using the physical markers,MR system 212 may output an indication that the physical markers may beremoved. In example where the physical markers are removed, MR system212 may maintain the registration of the virtual bone model to theobserved bone using virtual markers or any other suitable trackingtechnique.

In some examples, the practitioner may not remove the physical markersuntil a later point in the surgery. For instance, the practitioner maynot remove the physical markers until registration of the virtual modelto the observed bone is no longer required (e.g., after all virtualguidance that uses the registration has been displayed and correspondingsurgical steps have been completed).

In some examples, MR system 212 may be able to maintain the registrationbetween a virtual bone model and observed bone (e.g., glenoid, humerus,or other bone structure) throughout the procedure. However, in somecases, MR system 212 may lose, or otherwise be unable to maintain, theregistration between the virtual bone model and observed bone. Forinstance, MR system 212 may lose track of one of more of the markers(e.g., virtual, physical, or both). This loss may be the result of anynumber of factors including, but not limited to, body fluids (e.g.,blood) occluding the markers, the markers becoming dislodged (e.g., aphysical marker being knocked out of position), and the like. As such,MR system 212 may periodically determine whether registration has beenlost (2026).

In some examples, MR system 212 may determine that registration has beenlost where a confidence distance between a virtual point and acorresponding physical point exceeds a threshold confidence distance(e.g., a clinical value). MR system 212 may periodically determine theconfidence distance as a value that represents the accuracy of thecurrent registration. For instance, MR system 212 may determine that adistance between a virtual point and a corresponding physical point isless than 3mm.

In some examples, MR system 212 may output a representation of theconfidence distance. As one example, MR system 212 may causevisualization device 213 to display a numerical value of the confidencedistance. As another example, MR system 212 may cause visualizationdevice 213 to display a graphical representation of the confidencedistance relative to the threshold confidence distance (e.g., display agreen circle if the confidence distance is less than half of thethreshold confidence distance, display a yellow circle if the confidencedistance is between half of the threshold confidence distance and thethreshold confidence distance, and display a red circle if theconfidence distance greater than the threshold confidence distance).

In some examples, MR system 212 may utilize the same thresholdconfidence distance throughout a surgical procedure. For instance, MRsystem 212 may utilize a particular threshold confidence distance forall humeral and scapula work steps (e.g., described below with referenceto FIGS. 34-71). In some examples, MR system 212 may utilize differentthreshold confidence distances for various parts a surgical procedure.For instance, MR system 212 may utilize a first threshold confidencedistance for a first set of work steps and use a second thresholdconfidence distance (that is different than the first thresholdconfidence distance) for a first set of work steps for a second set ofwork steps.

Where registration has not been lost (“No” branch of 2026), MR system212 may continue to display virtual guidance (2024). However, where MRsystem 212 loses registration (“Yes” branch of 2026), MR system 212 mayperform one or more actions to re-register the virtual bone model to theobserved bone. As one example, MR system 212 may automatically attemptto perform the registration process without further action from thepractitioner. For instance, where physical markers have not beenremoved. MR system 212 may perform the registration process using thephysical markers. Alternatively, where the physical markers have beenremoved (or were never placed), MR system 212 may output a request forthe practitioner to place the physical markers. As such, MR system 212may be considered to periodically register the virtual model with theobserved bone.

In some examples, as opposed to automatically attempting re-registrationwhere registration is lost, MR system 212 may selectively performre-registration based on whether registration is still needed (2028). Insome examples, MR system 212 may determine that registration is stillneeded if additional virtual guidance will be displayed. Where MR system212 determines that registration is no longer needed (“No” branch of2028), MR system 212 may end the registration procedure.

As described above, MR system 212 may utilize any combination of virtualand physical markers to enable registration of virtual models tocorresponding observed structures. MR system 212 may use any of themarkers to perform an initial registration and, where needed, MR system212 may use any of the markers to perform a re-registration. The markersused for the initial registration may be the same as or may be differentthan the markers used for any re-registrations.

In some examples, to enhance the accuracy and quality of registration,during the initialization stage of the registration process, MR system212 can compute and display spatial constraints for user head pose andorientation. These constraints can be computed in real time and dependon the position of the user, and/or the orientation, and/or the distanceto the observed bone, and/or the depth camera characteristics. Forexample, MR system 212 may prompt the user to move closer to theobserved bone, to adjust the head position so that the user's gaze lineis perpendicular to the surface of interest of the observed bone, or tomake any other adjustments that can be useful to enhance theregistration process and which may depend on the particular surgicalapplication and/or the attributes of the particular anatomy of interestand/or the characteristics of the optical and depth sensors that areemployed in MR system 212.

In some examples, depth camera(s) 532 detect distance by using astructured light approach or time of flight of an optical signal havinga suitable wavelength. In general, the wavelength of the optical signalis selected so that penetration of the surface of the observed anatomyby the optical signal transmitted by depth camera(s) 532 is minimized.It should be understood, however, that other known or future developedtechniques for detecting distance also can be employed.

As discussed below, the registration techniques described herein may beperformed for any pair of virtual model and observed object. As oneexample, an MR system may utilize the registration techniques toregister a virtual model of a bone to an observed bone. For instance, anMR system may utilize the registration techniques to register a virtualmodel of a glenoid/humerus/ankle to a corresponding observedglenoid/humerus/ankle. As another example, an MR system may utilize theregistration techniques to register a virtual model of an implant to anobserved implant. An MR system may utilize the registration techniquesto register a virtual model of a tool to an observed tool. For instance,an MR system may utilize the registration techniques to register avirtual model of a drill to a corresponding observed drill.

In some examples, an MR system may perform the registration techniquesonce for a particular pair of a virtual model and an observed object(e.g., within a particular surgical procedure). For instance, an MRsystem may register a virtual model of a glenoid with an observedglenoid and utilize the registration to provide virtual guidance formultiple steps of a surgical procedure. In some examples, an MR systemmay perform the registration techniques multiple times for a particularpair of a virtual model and an observed object (e.g., within aparticular surgical procedure). For instance, an MR system may firstregister a virtual model of a glenoid with an observed glenoid andutilize the registration to provide virtual guidance for one or moresteps of a surgical procedure. Then, for example, after material hasbeen removed from the glenoid (e.g., via reaming), the MR system mayregister another virtual model of the glenoid (that accounts for theremoved material) with an observed glenoid and use the subsequentregistration to provide virtual guidance for one or more other steps ofthe surgical procedure.

Once registration is complete the surgical plan can be executed usingthe Augment Surgery mode of MR system 212. For example, FIG. 27illustrates an image perceptible to a user when in the augment surgerymode of a mixed reality system, according to an example of thisdisclosure. As shown in the example of FIG. 27, the surgeon canvisualize a virtually planned entry point 2700 and drilling axis 2702 onobserved bone structure 2200 and use those virtual images to assist withpositions and alignment of surgical tools. Drilling axis 2702 may alsobe referred to as a reaming axis, and provides a virtual guide fordrilling a hole in the glenoid for placement of a guide pin that willguide a reaming process. In some cases, drilling and placing the guidepin comprises a one-step process of drilling the guide pin into place(e.g., the guide pin may be “self-tapping”).

FIG. 28 illustrates an example of virtual images that a surgeon can seeof implant components via visualization device 213. Similarly, FIG. 29illustrates an example of virtual images that a surgeon can see ofimplant components. In the examples of FIG. 28 and FIG. 29, the surgeonalso can see virtual images of the implant components (e.g., the virtualbone model 1008), including the graft 1402, superimposed on observedbone structure 2200. Also, the surgeon can see the osteophytes and whichpart of the bone represents osteophytes.

The registration process may be used in conjunction with the virtualplanning processes and/or intra-operative guidance described elsewherein this disclosure. Thus, in one example, a virtual surgical plan isgenerated or otherwise obtained to repair an anatomy of interest of aparticular patient (e.g., the shoulder joint of the particular patient).In instances where the virtual surgical plan is obtained, anothercomputing system may generate the virtual surgical plan and an MR system(e.g., MR system 212) or other computing system obtains at least aportion of the virtual surgical plan from a computer readable medium,such as a communication medium or a non-transitory storage medium. Inthis example, the virtual surgical plan may include a 3D virtual modelof the anatomy of interest generated based on preoperative image dataand a prosthetic component selected for the particular patient to repairthe anatomy of interest. Furthermore, in this example, a user may use aMR system (e.g., MR system 212) to implement the virtual surgical plan.In this example, as part of using the MR system, the user may requestthe virtual surgical plan for the particular patient.

Additionally, the user may view virtual images of the surgical planprojected within a real environment. For example, MR system 212 maypresent 3D virtual objects such that the objects appear to reside withina real environment, e.g., with real anatomy of a patient, as describedin various examples of this disclosure. In other words, MR system 212may output, for viewing by a user, virtual images of the virtualsurgical plan projected within a real environment, where the virtualimages of the virtual surgical plan including the 3D virtual model ofthe anatomy of interest. In this example, the virtual images of thesurgical plan may include one or more of the 3D virtual model of theanatomy of interest, a 3D model of the prosthetic component selected torepair the anatomy of interest, and virtual images of a surgicalworkflow to repair the anatomy of interest. The virtual images of thesurgical workflow may include text, graphics, or animations indicatingone or more steps to perform as part of performing the surgery to repairthe anatomy of interest. Furthermore, in this example, the user mayregister the 3D virtual model with a real anatomy of interest of theparticular patient. The user may then implement the virtually generatedsurgical plan to repair the real anatomy of interest based on theregistration. In other words, in the augmented surgery mode, the usercan use the visualization device to align the 3D virtual model of theanatomy of interest with the real anatomy of interest.

In such examples, the MR system implements a registration processwhereby the 3D virtual model is aligned (e.g., optimally aligned) withthe real anatomy of interest. In this example, the user may register the3D virtual model with the real anatomy of interest without using virtualor physical markers. In other words, the 3D virtual model may be aligned(e.g., optimally aligned) with the real anatomy of interest without theuse of virtual or physical markers. The MR system may use theregistration to track movement of the real anatomy of interest duringimplementation of the virtual surgical plan on the real anatomy ofinterest. In some examples, the MR system may track the movement of thereal anatomy of interest without the use of tracking markers.

In some examples, as part of registering the 3D virtual model with thereal anatomy of interest, the 3D virtual model can be aligned (e.g., bythe user) with the real anatomy of interest and generate atransformation matrix between the 3D virtual model and the real anatomyof interest based on the alignment. The transformation matrix provides acoordinate system for translating the virtually generated surgical planto the real anatomy of interest. For instance, the registration processmay allow the user to view 3D virtual models of anatomical featuresassociated with steps of the virtual surgical plan projected oncorresponding real anatomical features of interest during the surgery onthe patient. The steps of the virtual surgical plan projected on thereal anatomy of interest include identification of an entry point forpositioning a prosthetic implant to repair the real anatomical featureof interest. In some examples, the alignment of the 3D virtual modelwith the real anatomy of interest may generate a transformation matrixthat may allow the user to view steps of the virtual surgical plan(e.g., identification of an entry point for positioning a prostheticimplant to repair the real anatomy of interest) projected on the realanatomy of interest.

In some examples, the registration process (e.g., the transformationmatrix generated using the registration process) may allow the user toimplement the virtual surgical plan on the real anatomy of interestwithout use of tracking markers. In some examples, aligning the 3Dvirtual model with the real anatomy of interest including positioning apoint of interest on a surface of the 3D virtual model at a location ofa corresponding point of interest on a surface of the real anatomy ofinterest and adjusting an orientation of the 3D virtual model so that avirtual surface normal at the point of interest is aligned with a realsurface normal at the corresponding point of interest. In some suchexamples, the point of interest is a center point of a glenoid.

With continued reference to FIG. 19, after performing the registrationprocess, the surgeon may perform a reaming axis drilling process (1906).During the reaming axis drilling process, the surgeon may drill areaming axis guide pin hole in the patient's glenoid to receive areaming guide pin. In some examples, at a later stage of the shouldersurgery, the surgeon may insert a reaming axis pin into the reaming axisguide pin hole. In some examples, the reaming axis pin may itself be thedrill bit that is used to drill the reaming axis guide pin hole (e.g.,the reaming axis pin may be self-tapping). Thus, in such examples, itmay be unnecessary to perform a separate step of inserting the reamingaxis pin. In some examples, an MR system (e.g., MR system 212, MR system1800A, etc.) may present a virtual reaming axis to help the surgeonperform the drilling in alignment with the reaming axis and therebyplace the reaming guide pin in the correct location and with the correctorientation.

The surgeon may perform the reaming axis drilling process in one ofvarious ways. For example, the surgeon may perform a guide-based processto drill the reaming axis pin hole. In the case, a physical guide isplaced on the glenoid to guide drilling of the reaming axis pin hole. Inother examples, the surgeon may perform a guide-free process, e.g., withpresentation of a virtual reaming axis that guides the surgeon to drillthe reaming axis pin hole with proper alignment. An MR system (e.g., MRsystem 212, MR system 1800A, etc.) may help the surgeon perform eitherof these processes to drill the reaming axis pin hole.

Furthermore, in the surgical process of FIG. 19, the surgeon may performa reaming axis pin insertion process (1908). During the reaming axis pininsertion process, the surgeon inserts a reaming axis pin into thereaming axis pin hole drilled into the patient's scapula. In someexamples, an MR system (e.g., MR system 212, MR system 1800A, etc.) maypresent virtual guidance information to help the surgeon perform thereaming axis pin insertion process.

After performing the reaming axis insertion process, the surgeon mayperform a glenoid reaming process (1910). During the glenoid reamingprocess, the surgeon reams the patient's glenoid. Reaming the patient'sglenoid may result in an appropriate surface for installation of aglenoid implant. In some examples, to ream the patient's glenoid, thesurgeon may affix a reaming bit to a surgical drill. The reaming bitdefines an axial cavity along an axis of rotation of the reaming bit.The axial cavity has an inner diameter corresponding to an outerdiameter of the reaming axis pin. After affixing the reaming bit to thesurgical drill, the surgeon may position the reaming bit so that thereaming axis pin is in the axial cavity of the reaming bit. Thus, duringthe glenoid reaming process, the reaming bit may spin around the reamingaxis pin. In this way, the reaming axis pin may prevent the reaming bitfrom wandering during the glenoid reaming process. In some examples,multiple tools may be used to ream the patient's glenoid. An MR system(e.g., MR system 212, MR system 1800A, etc.) may present virtualguidance to help the surgeon or other users to perform the glenoidreaming process. For example, the MR system may help a user, such as thesurgeon, select a reaming bit to use in the glenoid reaming process. Insome examples, the MR system present virtual guidance to help thesurgeon control the depth to which the surgeon reams the user's glenoid.In some examples, the glenoid reaming process includes a paleo reamingstep and a neo reaming step to ream different parts of the patient'sglenoid.

Additionally, in the surgical process of FIG. 19, the surgeon mayperform a glenoid implant installation process (1912). During theglenoid implant installation process, the surgeon installs a glenoidimplant in the patient's glenoid. In some instances, when the surgeon isperforming an anatomical shoulder arthroplasty, the glenoid implant hasa concave surface that acts as a replacement for the user's naturalglenoid. In other instances, when the surgeon is performing a reverseshoulder arthroplasty, the glenoid implant has a convex surface thatacts as a replacement for the user's natural humeral head. In thisreverse shoulder arthroplasty, the surgeon may install a humeral implantthat has a concave surface that slides over the convex surface of theglenoid implant. As in the other steps of the shoulder surgery of FIG.19, an MR system (e.g., MR system 212, MR system 1800A, etc.) maypresent virtual guidance to help the surgeon perform the glenoidinstallation process.

In some examples, the glenoid implantation process includes a process tofix the glenoid implant to the patient's scapula (1914). In someexamples, the process to fix the glenoid implant to the patient'sscapula includes drilling one or more anchor holes or one or more screwholes into the patient's scapula and positioning an anchor such as oneor more pegs or a keel of the implant in the anchor hole(s) and/orinserting screws through the glenoid implant and the screw holes,possibly with the use of cement or other adhesive. An MR system (e.g.,MR system 212, MR system 1800A, etc.) may present virtual guidance tohelp the surgeon with the process of fixing the glenoid implant theglenoid bone, e.g., including virtual guidance indicating anchor orscrew holes to be drilled or otherwise formed in the glenoid, and theplacement of anchors or screws in the holes.

Furthermore, in the example of FIG. 19, the surgeon may perform ahumerus preparation process (1916). During the humerus preparationprocess, the surgeon prepares the humerus for the installation of ahumerus implant. In instances where the surgeon is performing ananatomical shoulder arthroplasty, the humerus implant may have a convexsurface that acts as a replacement for the patient's natural humeralhead. The convex surface of the humerus implant slides within theconcave surface of the glenoid implant. In instances where the surgeonis performing a reverse shoulder arthroplasty, the humerus implant mayhave a concave surface and the glenoid implant has a correspondingconvex surface. As described elsewhere in this disclosure, an MR system(e.g., MR system 212, MR system 1800A, etc.) may present virtualguidance information to help the surgeon perform the humerus preparationprocess.

Furthermore, in the example surgical process of FIG. 19, the surgeon mayperform a humerus implant installation process (1918). During thehumerus implant installation process, the surgeon installs a humerusimplant on the patient's humerus. As described elsewhere in thisdisclosure, an MR system (e.g., MR system 212, MR system 1800A, etc.)may present virtual guidance to help the surgeon perform the humeruspreparation process.

After performing the humerus implant installation process, the surgeonmay perform an implant alignment process that aligns the installedglenoid implant and the installed humerus implant (1920). For example,in instances where the surgeon is performing an anatomical shoulderarthroplasty, the surgeon may nest the convex surface of the humerusimplant into the concave surface of the glenoid implant. In instanceswhere the surgeon is performing a reverse shoulder arthroplasty, thesurgeon may nest the convex surface of the glenoid implant into theconcave surface of the humerus implant. Subsequently, the surgeon mayperform a wound closure process (1922). During the wound closureprocess, the surgeon may reconnect tissues severed during the incisionprocess in order to close the wound in the patient's shoulder.

As mentioned elsewhere in this disclosure, a user interface of MR system212 may include workflow bar 1000. Workflow bar 1000 include iconscorresponding to workflow pages. In some examples, each workflow pagethat can be selected by the user (e.g., a surgeon) can include anAugment Surgery widget 1300 (such as that shown in FIG. 13), that, whenselected, launches an operational mode of MR system 212 in which a userwearing or otherwise using visualization device 213 can see the details(e.g., virtual images of details) of the surgical plan projected andmatched onto the patient bone and use the plan intraoperatively toassist with the surgical procedure. In general, the Augment Surgery modeallows the surgeon to register the virtual 3D model of the patient'sanatomy of interest (e.g., glenoid) with the observed real anatomy sothat the surgeon can use the virtual surgical planning to assist withimplementation of the real surgical procedure, as will be explained infurther detail below.

An example of Augment Surgery widget 1300 is shown in FIG. 13. Selectionof the widget 1300 initiates an augmented surgery mode of operation ofMR system 212. Upon initiation of the augmented surgery mode, 3D virtualbone model 1008 of the patient's relevant bone structure is registeredwith the observed bone structure (i.e., the patient's real bone) so thatdetails of a virtually planned procedure can be visualized andsuperimposed on the observed bone. For example, for a shoulderarthroplasty procedure, these details can include entry points, drillingaxes, osteophytes and cutting surfaces/planes, as examples. As shown inFIG. 13, Augment Surgery widget 1300 may permit a user to select, e.g.,with voice command keywords, whether the scapula is shown or not(Scapula ON/OFF) and, if shown, whether the scapula is shown as opaqueor transparent (Scapula Opaque/Transparent). In addition, the user mayselect, e.g., with voice command keywords, whether a glenoid reamingaxis is shown or not (Reaming Axis ON/OFF), whether everything is notshown (Everything Off), whether to rotate the displayed virtual objectsto the left or to the right (Rotation Left/Right), and whether to STOPthe rotation (Say STOP to Freeze).

As noted above, Augment Surgery widget 1300 may permit a user to selectwhether the scapula is shown as opaque or transparent (ScapulaOpaque/Transparent). In some examples, the user may use a voice command,hand gesture, or other type of command to select whether to show thescapula as opaque or transparent. When the user selects the element forcontrolling whether the scapula is opaque or transparent, visualizationdevice 213 may increase or decrease the opacity of the model of thescapula. In some examples, visualization device 213 may continuechanging the opacity of the model until visualization device 213receives an indication of user input, such as a voice command, to stopchanging the opacity. Changing the opacity of the model of the scapula,especially the glenoid portion of the scapula may help the user tobetter see the model of the scapula under different lighting conditions.

For a shoulder arthroplasty application, the registration process maystart by virtualization device 213 presenting the user with 3D virtualbone model 1008 of the patient's scapula and glenoid that was generatedfrom preoperative images of the patient's anatomy, e.g., by surgicalplanning system 102. The user can then manipulate 3D virtual bone model1008 in a manner that aligns and orients 3D virtual bone model 1008 withthe patient's real scapula and glenoid that the user is observing in theoperating environment. As such, in some examples, the MR system mayreceive user input to aid in the initialization and/or registration.However, discussed above, in some examples, the MR system may performthe initialization and/or registration process automatically (e.g.,without receiving user input to position the 3D bone model). For othertypes of arthroplasty procedures, such as for the knee, hip, foot, ankleor elbow, different relevant bone structures can be displayed as virtual3D images and aligned and oriented in a similar manner with thepatient's actual, real anatomy.

Regardless of the particular type of joint or anatomical structureinvolved, selection of the augment surgery mode initiates a procedurewhere 3D virtual bone model 1008 is registered with an observed bonestructure. In general, the registration procedure can be considered as aclassical optimization problem (e.g., either minimization ormaximization). For a shoulder arthroplasty procedure, known inputs tothe optimization (e.g., minimization) analysis are the 3D geometry ofthe observed patient's bone (derived from sensor data from thevisualization device 213, including depth data from the depth camera(s)532) and the geometry of the 3D virtual bone derived during the virtualsurgical planning state (such as by using the BLUEPRINT™ system). Otherinputs include details of the surgical plan (also derived during thevirtual surgical planning stage, such as by using the BLUEPRINT™system), such as the position and orientation of entry points, cuttingplanes, reaming axes and/or drilling axes, as well as reaming ordrilling depths for shaping the bone structure, the type, size and shapeof the prosthetic components, and the position and orientation at whichthe prosthetic components will be placed or, in the case of a fracture,the manner in which the bone structure will be rebuilt.

Upon selection of a particular patient from the welcome page of UI 522of MR system 212 (FIG. 5), the surgical planning parameters associatedwith that patient are connected with the patient's 3D virtual bone model1008, e.g., by one or more processors of visualization device 213. Inthe Augment Surgery mode, registration of 3D virtual bone model 1008(with the connected preplanning parameters) with the observed bone byvisualization device 213 allows the surgeon to visualize virtualrepresentations of the surgical planning parameters on the patient.

The optimization (e.g., minimization) analysis that is implemented toachieve registration of the 3D virtual bone model 1008 with the realbone generally is performed in two stages: an initialization stage andan optimization (e.g., minimization) stage. During the initializationstage, the user approximately aligns the 3D virtual bone model 1008 withthe patient's real bone, such as by using gaze direction, hand gesturesand/or voice commands to position and orient, or otherwise adjust, thealignment of the virtual bone with the observed real bone. Theinitialization stage will be described in further detail below. Duringthe optimization (e.g., minimization) stage, which also will bedescribed in detail below, an optimization (e.g., minimization)algorithm is executed that uses information from the optical camera(s)530 and/or depth camera(s) 532 and/or any other acquisition sensor(e.g., motion sensors 533) to further improve the alignment of the 3Dmodel with the observed anatomy of interest. In some examples, theoptimization (e.g., minimization) algorithm can be a minimizationalgorithm, including any known or future-developed minimizationalgorithm, such as an Iterative Closest Point algorithm or a geneticalgorithm as examples.

In this way, in one example, a mixed reality surgical planning methodincludes generating a virtual surgical plan to repair an anatomy ofinterest of a particular patient. The virtual surgical plan including a3D virtual model of the anatomy of interest is generated based onpreoperative image data and a prosthetic component selected for theparticular patient to repair the anatomy of interest. Furthermore, inthis example, a MR visualization system may be used to implement thevirtual surgical plan. In this example, using the MR system may compriserequesting the virtual surgical plan for the particular patient. Usingthe MR system also comprises viewing virtual images of the surgical planprojected within a real environment. For example, visualization device213 may be configured to present one or more 3D virtual images ofdetails of the surgical plan that are projected within a realenvironment, e.g., such that the virtual image(s) appear to form part ofthe real environment. The virtual images of the surgical plan mayinclude the 3D virtual model of the anatomy of interest, a 3D model ofthe prosthetic component, and virtual images of a surgical workflow torepair the anatomy of interest. Using the MR system may also includeregistering the 3D virtual model with a real anatomy of interest of theparticular patient. Additionally, in this example, using the MR systemmay include implementing the virtually generated surgical plan to repairthe real anatomy of interest based on the registration.

Furthermore, in some examples, the method comprises registering the 3Dvirtual model with the real anatomy of interest without using virtual orphysical markers. The method may also comprise using the registration totrack movement of the real anatomy of interest during implementation ofthe virtual surgical plan on the real anatomy of interest. The movementof the real anatomy of interest may be tracked without the use oftracking markers. In some instances, registering the 3D virtual modelwith the real anatomy of interest may comprise aligning the 3D virtualmodel with the real anatomy of interest and generating a transformationmatrix between the 3D virtual model and the real anatomy of interestbased on the alignment. The transformation matrix provides a coordinatesystem for translating the virtually generated surgical plan to the realanatomy of interest. In some examples, aligning may comprise virtuallypositioning a point of interest on a surface of the 3D virtual modelwithin a corresponding region of interest on a surface of the realanatomy of interest; and adjusting an orientation of the 3D virtualmodel so that a virtual surface shape associated with the point ofinterest is aligned with a real surface shape associated with thecorresponding region of interest. In some examples, aligning may furthercomprise rotating the 3D virtual model about a gaze line of the user.The region of interest may be an anatomical landmark of the anatomy ofinterest. The anatomy of interest may be a shoulder joint. In someexamples, the anatomical landmark is a center region of a glenoid.

In some examples, after a registration process is complete, a trackingprocess can be initiated that continuously and automatically verifiesthe registration between 3D virtual bone model 1008 and observed bonestructure 2200 during the Augment Surgery mode. During a surgery, manyevents can occur (e.g., patient movement, instrument movement, loss oftracking, etc.) that may disturb the registration between the 3Danatomical model and the corresponding observed patient anatomy or thatmay impede the ability of MR system 212 to maintain registration betweenthe model and the observed anatomy. Therefore, by implementing atracking feature, MR system 212 can continuously or periodically verifythe registration and adjust the registration parameters as needed. If MRsystem 212 detects an inappropriate registration (such as patientmovement that exceeds a threshold amount), the user may be asked tore-initiate the registration process.

In some examples, tracking can be implemented using one or more opticalmarkers, such as the marker 3010 shown in FIG. 30, that is fixed to aparticular location on the anatomy. MR system 212 monitors the opticalmarker(s) in order to track the position and orientation of the relevantanatomy in 3D space. If movement of the marker is detected, MR system212 can calculate the amount of movement and then translate theregistration parameters accordingly so as to maintain the alignmentbetween the 3D model and the observed anatomy without repeating theregistration process.

In other examples, tracking is markerless. For example, rather thanusing optical markers, MR system 212 implements markerless trackingbased on the geometry of the observed anatomy of interest. In someexamples, the markerless tracking may rely on the location of anatomicallandmarks of the bone that provide well-defined anchor points for thetracking algorithm. In situations or applications in which well-definedlandmarks are not available, a tracking algorithm can be implementedthat uses the geometry of the visible bone shape or other anatomy. Insuch situations, image data from optical camera(s) 530 and/or depthcameras(s) 532 and/or motion sensors 533 (e.g., IMU sensors) can be usedto derive information about the geometry and movement of the visibleanatomy. An example of a tracking algorithm that can be used formarkerless tracking is described in David J. Tan, et al., “6D ObjectPose Estimation with Depth Images: A Seamless Approach for RoboticInteraction and Augmented Reality,” arXiv:1709.01459v1 [cs,CV] (Sep. 5,2017), although any suitable tracking algorithm can be used. In someexamples, the markerless tracking mode of MR system 212 can include alearning stage in which the tracking algorithm learns the geometry ofthe visible anatomy before tracking is initiated. The learning stage canenhance the performance of tracking so that tracking can be performed inreal time with limited processing power.

FIG. 31 illustrates an example of a process flow 3100 for tracking in anaugment surgery mode of MR system 212, according to an example of thisdisclosure. The process of FIG. 31 may be performed by visualizationdevice 213 of MR system 212. At block 3102, a learning process isperformed during which the tracking algorithm learns the geometry of theanatomy of interest based on a virtual bone model. In some examples, thelearning is performed offline (i.e., before the surgery). At block 3104,tracking is initiated during the Augment Surgery Mode. At block 3106,movement of the anatomy of interest is continuously (or periodically)monitored. At block 3108, if detected movement exceeds a thresholdamount, the user may be prompted to re-initiate the registration processof FIG. 20A or FIG. 20B (block 3112). As discussed above, in someexamples, MR system 212 may automatically re-initiate and/or perform theregistration process if detected movement exceeds the threshold amount.Otherwise, the amount of movement is used to translate the registrationparameters, as needed (block 3110).

In some examples, marker and markerless tracking can both beimplemented. For example, optical markers can be used as a back-up tothe markerless tracking algorithm or as a verification of the trackingalgorithm. Further, the choice of implementing marker and/or markerlesstracking can be left to the discretion of the user or may depend on theparticular surgical procedure and the specific anatomical features thatare visible.

In some examples, to guide a surgeon in accordance with the surgicalplan, surgical instruments or tools (marker (e.g., visible, infrared,etc.) or markerless (e.g., tool geometry)) can be tracked to ensure thatinstrument pose and orientation are correct using any of the sametracking techniques described above. To guide the surgeon's use of thesurgical instruments, MR system 212 can display visible indicators orprovide other perceptible indications (e.g., vibrations, audible beeps,etc.) that prompt the surgeon to move the instrument in certaindirections. For example, MR system 212 can generate circles visible tothe surgeon that, when concentric, indicate that the tool is alignedaccording to the surgical plan.

On occasion, during a surgery, the surgeon may determine that there is aneed to modify the preoperative surgical plan. MR system 212 allows forintraoperative modifications to the surgical plan that then can beexecuted in the Augmented Surgery Mode. For instance, in some examples,the user can manipulate the user interface so that the user can view thevirtual surgical plan intraoperatively, including at least the 3Dvirtual bone anatomy of interest. In such examples, the user canmanipulate the user interface so that the user can modify the virtualsurgical plan intraoperatively. As an example, selection of the Planningpage on the workflow bar 1000 of the UI 522 shown in FIG. 10, whichallows the surgeon to view and manipulate 3D virtual bone model 1008 ofthe patient's anatomy and the prosthetic implant components 1010. UsingUI 522, the surgeon can rotate and translate the implant components 1010and change their type and size if desired. If changes are made, thevirtual surgical plan is automatically updated with the new parameters,which can then be connected with 3D virtual bone model 1008 when in theAugment Surgery mode. If registration has previously been completed withthe prior version of the virtual surgical plan, the planning parameterscan be updated. If the modifications to the virtual surgical planrequire the surgeon to repeat the registration process, MR system 212can prompt the surgeon to do so.

As discussed elsewhere in this disclosure, orthopedic surgicalprocedures may involve performing various work on a patient's anatomy.Some examples of work that may be performed include, but are notnecessarily limited to, cutting, drilling, reaming, screwing, adhering,and impacting. In general, it may be desirable for a practitioner (e.g.,surgeon, physician's assistant, nurse, etc.) to perform the work asaccurately as possible. For instance, if a surgical plan for implantinga prosthetic in a particular patient specifies that a portion of thepatient's anatomy is to be reamed at a particular diameter to aparticular depth, it may desirable for the surgeon to ream the portionof the patient's anatomy to as close as possible to the particulardiameter and to the particular depth (e.g., to increase the likelihoodthat the prosthetic will fit and function as planned and thereby promotea good health outcome for the patient).

In some examples, a surgeon may perform one of more work operations by“free hand” (i.e., by applying or otherwise using a tool withoutmechanical or visual guides/aids for the tool). For instance, as shownin FIGS. 32A-32C, in the course of a shoulder arthroplasty procedure, asurgeon may perform a surgical step of resection of humeral head 3204 ofhumerus 3200 by visually estimating (e.g., “eyeballing”) and markinganatomical neck 3202 of humerus 3200. The surgeon may then perform theresection of humeral head 3204 by guiding cutting tool 3206 (e.g., ablade of an oscillating saw) along the marked anatomical neck 3202 withthe surgeon's free hand, i.e., without mechanical or visual guidance.However, performing surgical steps involving these types of workoperations entirely by free hand may introduce unwanted error, possiblyundermining the results of the orthopedic surgical procedure.

In some examples, in the course of an orthopedic surgical procedure, asurgeon may perform one of more work operations, which also may bereferred to as surgical steps, with the assistance of a mechanicalguide. For instance, as shown in FIG. 33, a surgeon may attachmechanical guide 3300 on humerus 3200 prior to performing a resection ofhumeral head 3204 (e.g., as part of performing the humerus cut processof step 1902 of FIG. 19). The surgeon may adjust one or more componentsof mechanical guide 3300 such that top surface 3302 of mechanical guide3300 is co-planar with anatomic neck 3202 of humerus 3200 (for purposesof illustration, anatomic neck 3202 is illustrated as a broken line).After attaching mechanical guide 3300 to humeral head 3204 and adjustingthe mechanical guide, the surgeon may perform the resection of humeralhead 3204 by guiding a cutting tool (e.g., a blade of an oscillatingsaw) along top surface 3302. However, utilizing a mechanical guide maybe undesirable. As one example, attachment and/or adjustment of amechanical guide introduces additional time into a surgical procedure.As another example, the mechanical guide is an additional tool that mayresult in additional cost for the mechanical guide and/or additionaltime for sterilizing and tracking the mechanical guide (e.g., during theprocedure and during the pre-closing inventory).

In accordance with one or more techniques of this disclosure, avisualization system, such as MR visualization system 212, may beconfigured to display virtual guidance including one or more virtualguides for performing work on a portion of a patient's anatomy. Forinstance, the visualization system may display a virtual cutting planeoverlaid on an anatomic neck of the patient's humerus. In some examples,a user such as a surgeon may view real-world objects in a real-worldscene. The real-world scene may be in a real-world environment such as asurgical operating room. In this disclosure, the terms real andreal-world may be used in a similar manner. The real-world objectsviewed by the user in the real-world scene may include the patient'sactual, real anatomy, such as an actual glenoid or humerus, exposedduring surgery. The user may view the real-world objects via asee-through (e.g., transparent) screen, such as see-through holographiclenses, of a head-mounted MR visualization device, such as visualizationdevice 213, and also see virtual guidance such as virtual MR objectsthat appear to be projected on the screen or within the real-worldscene, such that the MR guidance object(s) appear to be part of thereal-world scene, e.g., with the virtual objects appearing to the userto be integrated with the actual, real-world scene. For example, thevirtual cutting plane/line may be projected on the screen of a MRvisualization device, such as visualization device 213, such that thecutting plane is overlaid on, and appears to be placed within, anactual, observed view of the patient's actual humerus viewed by thesurgeon through the transparent screen, e.g., through see-throughholographic lenses. Hence, in this example, the virtual cuttingplane/line may be a virtual 3D object that appears to be part of thereal-world environment, along with actual, real-world objects.

A screen through which the surgeon views the actual, real anatomy andalso observes the virtual objects, such as virtual anatomy and/orvirtual surgical guidance, may include one or more see-throughholographic lenses. The holographic lenses, sometimes referred to as“waveguides,” may permit the user to view real-world objects through thelenses and display projected holographic objects for viewing by theuser. As discussed above, an example of a suitable head-mounted MRdevice for visualization device 213 is the Microsoft HOLOLENS™ headset,available from Microsoft Corporation, of Redmond, Wash., USA. TheHOLOLENS™ headset includes see-through, holographic lenses, alsoreferred to as waveguides, in which projected images are presented to auser. The HOLOLENS™ headset also includes an internal computer, camerasand sensors, and a projection system to project the holographic contentvia the holographic lenses for viewing by the user. In general, theMicrosoft HOLOLENS™ headset or a similar MR visualization device mayinclude, as mentioned above, LCoS display devices that project imagesinto holographic lenses, also referred to as waveguides, e.g., viaoptical components that couple light from the display devices to opticalwaveguides. The waveguides may permit a user to view a real-world scenethrough the waveguides while also viewing a 3D virtual image presentedto the user via the waveguides. In some examples, the waveguides may bediffraction waveguides.

The presentation virtual guidance such as of a virtual cutting plane mayenable a surgeon to accurately resect the humeral head without the needfor a mechanical guide, e.g., by guiding a saw along the virtual cuttingplane displayed via the visualization system while the surgeon views theactual humeral head. In this way, a visualization system, such as MRsystem 212 with visualization device 213, may enable surgeons to performaccurate work (e.g., with the accuracy of mechanical guides but withoutthe disadvantages of using mechanical guides). This “guideless” surgerymay, in some examples, provide reduced cost and complexity.

The visualization system (e.g., MR system 212/visualization device 213)may be configured to display different types of virtual guides. Examplesof virtual guides include, but are not limited to, a virtual point, avirtual axis, a virtual angle, a virtual path, a virtual plane, and avirtual surface or contour. As discussed above, the visualization system(e.g., MR system 212/visualization device 213) may enable a user todirectly view the patient's anatomy via a lens by which the virtualguides are displayed, e.g., projected. The virtual guides may guide orassist various aspects of the surgery. For instance, a virtual guide mayguide at least one of preparation of anatomy for attachment of theprosthetic or attachment of the prosthetic to the anatomy.

The visualization system may obtain parameters for the virtual guidesfrom a virtual surgical plan, such as the virtual surgical plandescribed herein. Example parameters for the virtual guides include, butare not necessarily limited to: guide location, guide orientation, guidetype, guide color, etc.

The visualization system may display a virtual guide in a manner inwhich the virtual guide appears to be overlaid on an actual, realanatomical object of the patient, within a real-world environment, e.g.,by displaying the virtual guide(s) with actual, real-world patientanatomy (e.g., at least a portion of the patient's anatomy) viewed bythe user through holographic lenses. For example, the virtual guides maybe 3D virtual objects that appear to reside within the real-worldenvironment with the actual, real anatomical object.

The techniques of this disclosure are described below with respect to ashoulder arthroplasty surgical procedure. Examples of shoulderarthroplasties include, but are not limited to, reversed arthroplasty,augmented reverse arthroplasty, standard total shoulder arthroplasty,augmented total shoulder arthroplasty, and hemiarthroplasty. However,the techniques are not so limited, and the visualization system may beused to provide virtual guidance information, including virtual guidesin any type of surgical procedure. Other example procedures in which avisualization system, such as MR system 212, may be used to providevirtual guides include, but are not limited to, other types oforthopedic surgeries; any type of procedure with the suffix “plasty,”“stomy,” “ectomy,” “clasia,” or “centesis,”; orthopedic surgeries forother joints, such as elbow, wrist, finger, hip, knee, ankle or toe, orany other orthopedic surgical procedure in which precision guidance isdesirable.

A typical shoulder arthroplasty includes various work on a patient'sscapula and performing various work on the patient's humerus. The workon the scapula may generally be described as preparing the scapula(e.g., the glenoid cavity of the scapula) for attachment of a prosthesisand attaching the prosthesis to the prepared scapula. Similarly, thework on the humerus may generally be described as preparing the humerusfor attachment of a prosthesis and attaching the prosthesis to theprepared humerus. As described herein, the visualization system mayprovide guidance for any or all work performed in such an arthroplastyprocedure.

As discussed above, a MR system (e.g., MR system 212, MR system 1800A ofFIG. 18, etc.) may receive a virtual surgical plan for attaching aprosthetic to a patient and/or preparing bones, soft tissue or otheranatomy of the patient to receive the prosthetic. The virtual surgicalplan may specify various work to be performed and various parameters forthe work to be performed. As one example, the virtual surgical plan mayspecify a location on the patient's glenoid for performing reaming and adepth for the reaming. As another example, the virtual surgical plan mayspecify a surface for resecting the patient's humeral head. As anotherexample, the virtual surgical plan may specify locations and/ororientations of one or more anchorage locations (e.g., screws, stems,pegs, keels, etc.).

In some examples, MR system 212 may provide virtual guidance to assist asurgeon in performing work on a patient's humerus. As shown in FIGS.34-41, MR system 212 may provide virtual guidance to assist a surgeon inpreparing and removing a bone graft from a head of the patient'shumerus. As shown in FIGS. 42A-49, MR system 212 may provide virtualguidance to assist a surgeon in humeral preparation, such as cutting toremove all or a portion of the humeral head. FIG. 50 is a conceptualdiagram illustrating MR system 212 providing virtual guidance forattaching an implant to a humerus, in accordance with one or moretechniques of this disclosure. A tool may be used to attach the implantto humerus 3200. For instance, the surgeon may utilize handle 4802 toinsert prosthesis 5000 into the prepared humerus 3200. In some examples,one or more adhesives (e.g., glue, cement, etc.) may be applied toprosthesis 5000 and/or humerus 3200 prior to insertion. As shown in FIG.50, MR system 212 may provide virtual guidance to assist a surgeon inhumeral implant positioning, such as preparation of the humerus toreceive an implant and positioning of the implant within the humerus.

In some examples, MR system 212 may provide virtual guidance to assist asurgeon in performing work on a patient's scapula. As shown in FIGS.51-62, the MR system may provide virtual guidance to assist a surgeon inscapula preparation (e.g., as part of performing the reaming axisdrilling process of step 1906 of FIG. 19, as part of performing thereaming axis guide pin insertion process of step 1908 of FIG. 19, and/oras part of performing the glenoid remaining process of step 1910 of FIG.19). As shown in FIGS. 63-65, the MR system may provide virtual guidanceto assist a surgeon in scapula implant positioning (e.g., as part ofperforming the glenoid implant installation process of step 1912 of FIG.19).

Many different techniques may be used to prepare a humerus forprosthesis attachment and to perform actual prosthesis attachment.Regardless of the technique used, MR system 212 may provide virtualguidance to assist in one or both of the preparation and attachment. Assuch, while the following techniques are examples in which MR system 212provides virtual guidance, MR system 212 may provide virtual guidancefor other techniques.

In an example technique, the work steps include resection of a humeralhead, creating a pilot hole, sounding, punching, compacting, surfacepreparation, with respect to the humerus, and attaching an implant tothe humerus. Additionally, in some techniques, the work steps mayinclude bone graft work steps, such as installation of a guide in ahumeral head, reaming of the graft, drilling the graft, cutting thegraft, and removing the graft, e.g., for placement with an implant foraugmentation of the implant relative to a bone surface such as theglenoid.

A surgeon may perform one or more steps to expose a patient's humerus.For instance, the surgeon may make one or more incisions to expose theupper portion of the humerus including the humeral head. The surgeon mayposition one or more retractors to maintain the exposure. In someexamples, MR system 212 may provide guidance to assist in the exposureof the humerus, e.g., by making incisions, and/or placement ofretractors.

FIGS. 34 and 35 are conceptual diagrams illustrating an MR systemproviding virtual guidance for installation of a mechanical guide in ahumeral head, in accordance with one or more techniques of thisdisclosure. It is noted that, for purposes of illustration, thesurrounding tissue and some bone is omitted from FIGS. 34 and 35, andother figures. As shown in FIG. 34, MR system 212 may display virtualaxis 3400 on humeral head 3204 of humerus 3200. FIG. 34 and subsequentfigures illustrate what the surgeon, or other user, would see whenviewing via visualization device 213. In particular, when viewing viavisualization device 213, the surgeon would see a portion of humerus3200 and virtual axis 3400 (and/or other virtual guidance) overlaid onthe portion of humerus 3200.

To display virtual axis 3400, MR system 212 may determine a location ona virtual model of humerus 3200 at which a guide is to be installed. MRsystem 212 may obtain the location from a virtual surgical plan (e.g.,the virtual surgical plan described above as generated by virtualplanning system 202). The location obtained by MR system 212 may specifyone or both of coordinates of a point on the virtual model and a vector.The point may be the position at which the guide is to be installed andthe vector may indicate the angle/slope at which the guide is to beinstalled. As such, MR system 212 may display a virtual drilling axishaving parameters obtained from the virtual surgical plan, the virtualdrilling axis configured to guide drilling of one or more holes in theglenoid (e.g., for attachment of a guide pin to the scapula).

As discussed above, the virtual model of humerus 3200 may be registeredwith humerus 3200 such that coordinates on the virtual modelapproximately correspond to coordinates on humerus 3200. As such, bydisplaying virtual axis 3400 at the determined location on the virtualmodel, MR system 212 may display virtual axis 3400 at the plannedposition on humerus 3200.

The surgeon may attach a physical guide using the displayed virtualguidance. For instance, where the guide is a guide pin with aself-tapping threaded distal tip, the surgeon may align the guide pinwith the displayed virtual axis 3400 and utilize a drill or otherinstrument to install the guide pin. In some examples, MR system 212 maydisplay depth guidance information to enable the surgeon to install theguide pin to a planned depth. Examples of depth guidance information arediscussed in further detail herein with reference to FIGS. 66-68.

FIG. 35 is a conceptual diagram illustrating guide 3500 as installed inhumeral head 3204. Guide 3500 may take the form of an elongated pin tobe mounted in a hole formed in the humeral head. As shown in FIGS. 34and 35, by displaying virtual axis 3400, a surgeon may install guide3500 at the planned position on humeral head 3204. In this way, MRsystem 212 may enable the installation of a guide without the need foran additional mechanical guide.

As discussed above, MR system 212 may provide virtual guidance, such asvirtual markers, to assist the surgeon in the installation of the guidepin. For instance, in the example of FIG. 34, MR system 212 may displayvirtual axis 3400 to assist the surgeon in the installation of the guidepin. Other examples of virtual markers that MR system 212 may displayinclude, but are not limited to axes, points, circles, rings, polygons,X shapes, crosses, or any other shape or combination of shapes. MRsystem 212 may display the virtual markers as static or with variousanimations or other effects.

FIGS. 36A-36D illustrate examples of virtual markers that MR system 212may display. FIG. 36A illustrates an example in which MR system 212displays virtual marker 3600A as a point. FIG. 36B illustrates anexample in which MR system 212 displays virtual marker 3600B as across/X shape. FIG. 36C illustrates an example in which MR system 212displays virtual marker 3600C as a reticle. FIG. 36D illustrates anexample in which MR system 212 displays virtual marker 3600D ascombination of a reticle and an axis.

As discussed above, in some examples, MR system 212 may display thevirtual markers with various animations or other effects. As oneexample, MR system 212 may display a virtual marker as a reticle havinga rotating ring. As another example, MR system 212 may display a virtualmarker as a flashing cross/X shape.

MR system 212 may display the virtual markers with particular colors.For instance, in some examples, MR system 212 may preferably display thevirtual markers in a color other than red, such as green, blue, yellow,etc. Displaying the virtual markers in a color or colors other than redmay provide one or more benefits. For instance, as blood appears red andblood may be present on or around the anatomy of interest, a red coloredvirtual marker may not be visible.

The use of the various types of virtual markers described above is notlimited to installation of the guide pin. For instance, MR system 212may display any of the virtual markers described above to assist thesurgeon in performing any work. As one example, MR system 212 maydisplay any of the virtual markers described above to assist the surgeonin performing any work on humerus 3200. As another example, MR system212 may display any of the virtual markers described above to assist thesurgeon in performing any work on scapula 5100.

FIGS. 37 and 38 are conceptual diagrams illustrating an MR systemproviding virtual guidance for reaming of a graft in a humeral head, inaccordance with one or more techniques of this disclosure. As shown inFIGS. 37 and 38, graft reaming tool 3700 may be used to ream the surfaceof humeral head 3204 and cut outline 3800 (shown in FIG. 38).

The surgeon may connect graft reaming tool 3700 to a drill or otherinstrument and MR system 212 may display virtual guidance to assist inreaming the surface of humeral head 3204 and cutting outline 3800. Forinstance, MR system 212 may display depth guidance to enable the surgeonto ream the surface of humeral head 3204 and cut outline 3800 to atarget depth (e.g., depth guidance similar to the depth guidancediscussed below with reference to FIGS. 66-68). As another example, MRsystem 212 may provide targeting guidance. For instance, MR system 212may display one or both of a virtual marker that identifies a centerpoint or prescribed axis of the reaming (e.g., as discussed above withreference to FIGS. 36A-36D) and/or an indication of whether graftreaming tool 3700 is aligned with the prescribed axis. As shown in FIG.38, the surgeon may remove graft reaming tool 3700 from guide 3500 afterperforming the reaming.

In this example, graft reaming tool 3700 may be a cannulated reamingtool configured to be positioned and/or guided by a guide pin, such asguide 3500. In other examples, graft reaming tool 3700 may not becannulated and may be guided without the assistance of a physical guidepin. For instance, MR system 212 may provide virtual guidance (e.g.,depth guidance and/or targeting guidance such as a displayed virtualmarker) to enable a surgeon to ream a graft from humeral head 3204without the use of guide 3500. As such, MR system 212 may display avirtual reaming axis having parameters (e.g., position, size, and/ororientation relative to the virtual model of the scapula) obtained fromthe virtual surgical plan. The displayed virtual reaming axis may beconfigured to guide reaming of the humeral head and/or a graft from thehumeral head.

FIGS. 39 and 40 are conceptual diagrams illustrating an MR system, suchas MR system 212, providing virtual guidance for drilling a graft in ahumeral head, in accordance with one or more techniques of thisdisclosure. As shown in FIGS. 39 and 40, drill bit 3900 may be used todrill central hole 4000 in humeral head 3204.

The surgeon may connect drill bit 3900 to a drill or other instrumentand MR system 212 may display virtual guidance to assist in the creationof central hole 4000. For instance, MR system 212 may display depthguidance to enable the surgeon to drill central hole 4000 to a targetdepth (e.g., depth guidance similar to the depth guidance discussedbelow with reference to FIGS. 66-68). As another example, MR system 212may provide targeting guidance. For instance, MR system 212 may displayone or both of a virtual marker that identifies a center point orprescribed axis of the drilling (e.g., as discussed above with referenceto FIGS. 36A-36D) and/or an indication of whether drill bit 3900 is on aprescribed axis.

In this example, drill bit 3900 may be a cannulated reaming toolconfigured to be positioned and/or guided by a guide pin, such as guide3500. In other examples, drill bit 3900 may not be cannulated and may beguided without the assistant of a physical guide pin. For instance, MRsystem 212 may provide virtual guidance (e.g., depth guidance and/ortargeting guidance such as a virtual marker) to enable a surgeon todrill central hole 4000 without the use of guide 3500.

FIG. 41 is a conceptual diagram illustrating an MR system providingvirtual guidance for cutting of a graft in a humeral head, in accordancewith one or more techniques of this disclosure. As shown in FIG. 41, thereaming and drilling work steps discussed above may result in graft 4102having a toroid shape with the bottom surface still attached to humerus3200. The surgeon may use a tool, such as oscillating saw 4104, to cutgraft 4102 from humerus 3200.

MR system 212 may display virtual guidance to assist in the cuttingprocess. As one example, MR system 212 may provide targeting guidance.For instance, MR system 212 may display a virtual marker such as virtualcutting surface 4100 (e.g., a virtual cutting plane or any of thevirtual markers discussed above with reference to FIGS. 36A-36D) thatindicates where the surgeon should cut using the tool and/or anindication of whether the tool (e.g., oscillating saw 4104) is alignedwith the virtual cutting surface. MR system 212 may obtain the positionand orientation of the virtual cutting plane from the virtual surgicalplan. As such, MR system 212 may display a virtual cutting plane havingparameters (e.g., position, size, and/or orientation relative to thevirtual model of the humeral head) obtained from a virtual surgical planthat guides cutting of a graft from a humeral head. As shown in FIG. 41,MR system 212 may display virtual cutting surface 4100 on a planeparallel to, or the same as, the bottom of graft 4102. As anotherexample, MR system 212 may display a virtual model of graft 4102. Forinstance, MR system 212 may display an outline of graft 4102. As anotherexample, MR system 212 may provide depth guidance (e.g., depth guidancesimilar to the depth guidance discussed below with reference to FIGS.66-68). For instance, MR system 212 may display depth guidance to enablethe surgeon to cut to a target depth.

The surgeon may utilize the graft for any purpose. For instance, thesurgeon may utilize the graft to fill empty space between a prosthesisan a glenoid of the patient and/or provide/increase an offset whenattaching a prosthesis to a glenoid of the patient.

In order to prepare the humerus for implantation of the prosthesis, thesurgeon may resect, cut, or otherwise remove the humeral head. SeveralMR assisted techniques for humeral head resection are contemplated,including techniques involving cutting the humeral head with removal ofa graft and cutting the humeral head without removal of a graft. In afirst example technique, MR system 212 may display a virtual cuttingsurface, such as a virtual cutting plane, that guides the surgeon inresecting the humeral head, e.g., without taking a graft. In this case,there may be no need for a mechanical guide, making the procedure lesscomplex and possibly less costly, while still maintaining accuracy.Further details of the first example technique are discussed below withreference to FIGS. 42A-42C. In a second example technique, MR system 212may display a virtual axis that guides the surgeon in installing aphysical guide, i.e., mechanical guide, on the humerus, which thenguides the surgeon in resecting the humeral head. Further details of thesecond example technique are discussed below with reference to FIG. 43.

FIGS. 42A-42C are conceptual diagrams illustrating an MR systemproviding virtual guidance for resection of a humeral head, inaccordance with one or more techniques of this disclosure. As shown inFIGS. 42A and 42B, MR system 212 may display virtual cutting plane 4200at a planned position on humerus 3200. To display virtual cutting plane4200, MR system 212 may determine a location on a virtual model ofhumerus 3200 at which humeral head 3204 is to be resected. MR system 212may obtain the location from a virtual surgical plan (e.g., the virtualsurgical plan described above). As such, MR system 212 may display avirtual cutting surface (e.g., cutting plane) having parameters (e.g.,position, size, and/or orientation relative to the virtual model of thehumerus) obtained from the virtual surgical plan that guides resectionof a portion of a head of the humerus.

As discussed above, a virtual model of humerus 3200 may be registeredwith humerus 3200 such that coordinates on the virtual modelapproximately correspond to coordinates on humerus 3200. As such, bydisplaying virtual cutting plane 4200 at the determined location on thevirtual model, MR system 212 may display virtual cutting plane 4200 atthe planned position on humerus 3200.

The surgeon may resect humeral head 3204 using the displayed virtualguidance. For instance, the surgeon may utilize oscillating saw 4104 toresect humeral head 3204 by cutting along virtual cutting plane 4200. Insome examples, MR system 212 may display targeting guidance to indicatewhether the tool (e.g., oscillating saw 4104) is on the prescribedplane.

FIG. 43 is a conceptual diagram illustrating a physical guide forhumeral head resection that is positioned using virtual guidance, inaccordance with one or more techniques of this disclosure. As discussedabove, in the second example technique, MR system 212 may display avirtual axis that guides the surgeon in installing a physical guide,which guides the surgeon in resecting the humeral head. For instance, MRsystem 212 may display a virtual marker, such as a virtual axis, usingtechniques similar to those discussed above with reference to FIGS. 34and 35. The surgeon may use the virtual axis to guide installation ofphysical guide 3500 (e.g., a guide pin). As such, MR system 212 maydisplay a virtual drilling axis having parameters (e.g., position, size,and/or orientation relative to the virtual model of the humerus)obtained from the virtual surgical plan that guides attachment of aguide pin to the humerus. As discussed above, the guide pin may beconfigured to guide attachment of a resection guide to the humerus.

The surgeon may use guide 3500 to assist in the installation ofresection guide 4300 (e.g., the guide pin may be configured to guideattachment of a resection guide to the humerus). In general, resectionguide 4300 may be a physical assembly configured to physically guide atool (e.g., an oscillating saw) for resecting a humeral head. In theexample of FIG. 43, resection guide 4300 includes plates 4302A and 4302B(collectively, “plates 4302”), upper plate 4308, adjustment screws 4306Aand 4306B (collectively, “adjustment screws 4306”), and guide receiver4310.

Guide receiver 4310 may be sized to accept guide 3500 such thatresection guide 4300 may be passed over guide 3500. Plates 4302 defineslot 4304, which may be sized to receive and guide a physically guide atool (e.g., an oscillating saw) between plates 4302 and across cuttingplane 4312. Upper plate 4308 may be configured to rest against a top ofhumeral head 3204 (either native or after work has been performed toremove a graft). Adjustment screws 4306 may be collectively orindependently adjusted to position plates 4302, and thus cutting plane4312, relative to upper plate 4308.

MR system 212 may provide virtual guidance to assist in the positioningof resection guide 4300. As one example, MR system 212 may display avirtual cutting plane at the desired location of cutting plane 4312. Thesurgeon may adjust adjustment screws 4306 until slot 4304 is alightedwith the virtual cutting plane. In some examples, MR system 212 mayprovide guidance as to which of adjustment screws 4306 is to betightened or loosened. Once resection guide 4300 is properly configured(e.g., slot 4304 is alighted with the virtual cutting plane), thesurgeon may operate a tool to resect humeral head 3204.

FIGS. 44 and 45 are conceptual diagrams illustrating an MR systemproviding virtual guidance for creating a pilot hole in a humerus, inaccordance with one or more techniques of this disclosure. As shown inFIGS. 44 and 45, starter awl 4400 may be used to create a pilot holein-line with a humeral canal at a hinge point of the resection.

MR system 212 may provide virtual guidance to assist in the creation ofthe pilot hole. As one example, MR system 212 may display targetingguidance, such as a virtual marker (e.g., virtual point 4402) thatrepresents the location at which the surgeon should create the pilothole. For instance, MR system 212 may display a virtual axis havingparameters (e.g., position, size, and/or orientation relative to thevirtual model of the humerus) obtained from the virtual surgical planthat guides creation of a pilot hole in the humerus after a head of thehumerus has been resected. As another example MR system 212 may displaydepth guidance, e.g., the depth guidance discussed below with referenceto FIGS. 66-68, to assist the surgeon in creating the pilot hole to aprescribed depth.

FIG. 46 is a conceptual diagram illustrating an MR system providingvirtual guidance for sounding a humerus, in accordance with one or moretechniques of this disclosure. As shown in FIG. 46, sounder 4600 may beused to determine an upper size limit of a distal portion of humerus3200. In some examples, as discussed herein, multiple sounders ofdifferent sizes may be used to the upper size limit.

MR system 212 may provide virtual guidance to assist in the sounding. Asone example, MR system 212 may display virtual targeting guidance forsounder 4600. For instance, MR system 212 may display a virtual marker(e.g., as discussed above with reference to FIGS. 36A-36D) thatindicates where sounder 4600 should be inserted.

FIG. 47 is a conceptual diagram illustrating an MR system providingvirtual guidance for punching a humerus, in accordance with one or moretechniques of this disclosure. As shown in FIG. 47, the surgeon mayattach punch template 4700 to sounder 4600 (or the final sounderdetermined during the sounding step). The surgeon may then place punch4702 into template 4700 until punch 4702 bottoms out on template 4700.The surgeon may then remove the scored bone by pulling sounder 4600,template 4700, and punch 4702 out of humerus 3200.

MR system 212 may provide virtual guidance to assist in the punching. Asone example, MR system 212 may display an indication of whether punch4702 is properly positioned in template 4700. For instance, where punch4702 is properly positioned in template 4700, MR system 212 may displaya virtual marker that indicates proper position (e.g., a checkmark).Similarly, where punch 4702 is not properly positioned in template 4700,MR system 212 may display a virtual marker that indicates improperposition (e.g., an X).

FIG. 48 is a conceptual diagram illustrating an MR system providingvirtual guidance for compacting a humerus, in accordance with one ormore techniques of this disclosure. As shown in FIG. 48, compactor 4800may be attached to handle 4802 and inserted into humerus 3200. In someexamples, multiple compactors may be used. For instance, the surgeon maybegin with a compactor three sizes below a size of the final sounder andcompact sequentially until satisfactory fixation is achieved.Satisfactory fixation can be assessed by a slight torque motion ofhandle 4802. Compactor 4800 should not move within the humerus duringthis test if satisfactory fixation has been achieved.

MR system 212 may provide virtual guidance to assist in the compacting.As one example, MR system 212 may display indication of whethersatisfactory fixation has been achieved. For instance, where MR system212 determines that satisfactory fixation has been achieved, MR system212 may display a virtual marker that indicates satisfactory fixation(e.g., a checkmark). Similarly, where MR system 212 determines thatsatisfactory fixation has not been achieved, MR system 212 may display avirtual marker that indicates unsatisfactory fixation (e.g., an X).

The surgeon may disconnect compactor 4800 (e.g., the final compactor)from handle 4802. The surgeon may then perform one or more surfacepreparation steps.

FIG. 49 is a conceptual diagram illustrating an MR system providingvirtual guidance for preparing a surface of a humerus, in accordancewith one or more techniques of this disclosure. As shown in FIG. 49, thesurgeon may use surface planner 4900 to prepare a surface of humerus3200 (e.g., to ensure a flat resection true to the prosthesis).

MR system 212 may provide virtual guidance to assist in the surfacepreparation. For instance, MR system 212 may provide targeting guidance(e.g., similar to the targeting guidance discussed below with referenceto FIGS. 66-68) to aid the surgeon in keeping surface planner 4900 on aplanned/prescribed axis.

Many different techniques may be used to prepare a scapula forprosthesis attachment and to perform actual prosthesis attachment.Regardless of the technique used, MR system 212 may provide virtualguidance to assist in one or both of the preparation and attachment. Assuch, while the following techniques are examples in which MR system 212provides virtual guidance, MR system 212 may provide virtual guidancefor other techniques.

In an example technique, the surgical procedure steps includeinstallation of a guide in a glenoid of the scapula, reaming theglenoid, creating a central hole in the glenoid, creating additionalanchorage positions in the glenoid, and attaching an implant to theprepared glenoid. As a guide pin is used, the example technique may beconsidered a cannulated technique. However, the techniques are similarlyapplicable to non-cannulated techniques.

A surgeon may perform one or more steps to expose a patient's glenoid.For instance, with the patient's arm abducted and internally rotated,the surgeon may make one or more incisions to expose the glenoid. Thesurgeon may position one or more retractors to maintain the exposure. Insome examples, MR system 212 may provide guidance to assist in theexposure and/or placement of retractors.

FIG. 51 is a conceptual diagram illustrating an MR system providingvirtual guidance to a user for installation of a guide in a glenoid of ascapula, in accordance with one or more techniques of this disclosure.As shown in FIG. 51, MR system 212 may display virtual guidance, e.g.,in the form of virtual axis 5104, on glenoid 5102 of scapula 5100. Todisplay virtual axis 5104, MR system 212 may determine a location on avirtual model of glenoid 5102 at which a guide is to be installed. MRsystem 212 may obtain the location from a virtual surgical plan (e.g.,the virtual surgical plan described above). The location obtained by MRsystem 212 may specify one or both of coordinates of a point on thevirtual model and a vector. The point may be the position at which theguide is to be installed and the vector may indicate the angle/slope atwhich the guide is to be installed. As such, MR system 212 may display avirtual reaming axis having parameters (e.g., position, size, and/ororientation relative to the virtual model of the scapula) obtained fromthe virtual surgical plan. The displayed virtual reaming axis may beconfigured to guide reaming of the glenoid.

As discussed above, the virtual model of glenoid 5102 may be registeredwith glenoid 5102 such that coordinates on the virtual modelapproximately correspond to coordinates on glenoid 5102. As such, bydisplaying virtual axis 5104 at the determined location on the virtualmodel, MR system 212 may display virtual axis 5104 at the plannedposition on glenoid 5102.

As also discussed above, the virtual model of glenoid 5102 may beselectively displayed after registration. For instance, after thevirtual model of glenoid 5102 is registered with glenoid 5102, MR system212 may cease displaying of the virtual model. Alternatively, MR system212 may continue to display the virtual model overlaid on glenoid 5102after registration. The display of the virtual model may be selective inthat the surgeon may activate or deactivate display of the virtualmodel.

MR system 212 may display the virtual model and/or virtual guides withvarying opacity (e.g., transparency). The opacity may be adjustedautomatically, manually, or both. As one example, the surgeon mayprovide user input to MR system 212 to manually adjust the opacity ofthe virtual model and/or virtual guides. As another example, MR system212 may automatically adjust the opacity based on an amount of light inthe viewing field (e.g., amount of light where the surgeon is looking).For instance, MR system 212 may adjust the opacity (e.g., increase thetransparency) of the virtual model and/or virtual guides to positivelycorrelate with the amount of light in the viewing field (e.g., brighterlight results in increased opacity/decreased transparency and dimmerlight results in decreased opacity/increased transparency).

The surgeon may attach a physical guide using the displayed virtualguidance. As one example, where the guide is a guide pin with aself-tapping threaded distal tip, the surgeon may align the guide pinwith the displayed virtual axis 5104 and utilize a drill or otherinstrument to install the guide pin. As another example, where the guideis a guide pin without a self-tapping tip, the surgeon may align a drillbit of a drill with the displayed virtual axis 5104 and operate thedrill to form a hole to receive the guide pin and then install the guidepin in the hole. In some examples, MR system 212 may display depthguidance information to enable the surgeon to install the guide pin to aplanned depth. Examples of depth guidance information are discussed infurther detail herein with reference to FIG. 66.

FIG. 52 is a conceptual diagram illustrating guide 5200, i.e., a guidepin in this example, as installed in glenoid 5102. As shown in FIGS. 51and 52, by displaying virtual axis 5104, a surgeon may drill inalignment with the virtual axis, which may be referred to as a reamingaxis, and thereby form a hole for installation of guide 5200 at theplanned position on glenoid 5102. In this way, MR system 212 may enablethe installation of a guide without the need for an additionalmechanical guide.

FIG. 53 is a conceptual diagram illustrating an MR system providingvirtual guidance for reaming a glenoid, in accordance with one or moretechniques of this disclosure. As shown in FIG. 53, reaming tool 5300may be used to ream the surface of glenoid 5102. In this example,reaming tool 5300 may be a cannulated reaming tool configured to bepositioned and/or guided by a guide pin, such as guide 5200. Forexample, the shaft of cannulated reaming tool may receive guide 5200such that the tool shaft is mounted substantially concentrically withthe pin. In other examples, reaming tool 5300 may not be cannulated andmay be guided without the assistance of a physical guide pin.

The surgeon may attach reaming tool 5300 to guide 5200 (e.g., insertproximal tip of guide 5200 into reaming tool 5300), and attach a drillor other instrument to rotate reaming tool 5300. To perform the reaming,the surgeon may rotate reaming tool 5300 to advance reaming tool 5300down guide 5200 until reaming is complete.

MR system 212 may display virtual guidance to assist in the reamingprocess. As one example MR system 212 may provide depth guidance. Forinstance, MR system 212 may display depth guidance to enable the surgeonto ream to a target depth. As another example, MR system 212 may providetargeting guidance. For instance, MR system 212 may display anindication of whether reaming tool 5300 is aligned with a virtualreaming axis.

While described herein as a single reaming step, the surgery may includemultiple reaming steps. The various reaming steps may use the sameaxis/guide pin or may use different axes/guide pins. In examples wheredifferent reaming steps use different axes, MR system 212 may providevirtual guidance for reaming using the different axes.

FIGS. 54 and 55 are conceptual diagrams illustrating an MR systemproviding virtual guidance for creating a central hole in a glenoid(e.g., post-reaming), in accordance with one or more techniques of thisdisclosure. As shown in FIGS. 54 and 55, drill bit 5400 may be used todrill central hole 5500 in glenoid 5102. In this example, drill bit 5400may be a cannulated drill bit configured to be positioned and/or guidedby a guide pin, such as guide 5200. In other examples, drill bit 5400may not be cannulated and may be guided without the assistance of aphysical guide pin. For instance, MR system 212 may provide virtualguidance (e.g., any combination of virtual markers, depth guidance,and/or targeting guidance discussed below with reference to FIGS. 66-68)to enable a surgeon to drill glenoid 5102 without the use of guide 5200.As discussed in further detail below, central hole 5500 may facilitatethe attachment of a prosthesis to glenoid 5102, e.g., via one or moreanchors.

MR system 212 may display virtual guidance to assist in the creation ofcentral hole 5500. For instance, MR system 212 may display depthguidance to enable the surgeon to drill central hole 5500 to a targetdepth. As another example, MR system 212 may provide targeting guidance(e.g., any combination of virtual markers and/or targeting guidancediscussed below with reference to FIGS. 66-68). For instance, MR system212 may display an indication of whether drill bit tool 5400 is on aprescribed axis selected to form the central hole 5500 at a properposition at with a proper orientation.

In addition to a central hole (e.g., central hole 5500), it may bedesirable for the surgeon to create additional anchorage positions inthe glenoid. This additional anchorage positions may improve thefixation between the prosthesis and the glenoid. For instance, theadditional anchorage positions may provide anti-rotation support betweenthe prosthesis and the glenoid. Several different styles of anchoragemay be used, depending on the type of prosthesis to be installed. Someexamples of anchorage include, but are not necessarily limited to, keeland pegged anchors. However, the virtual guidance techniques discussedherein may be applicable to any type of anchorage. Example MR guidancefor keel type anchorage is discussed below with reference to FIGS.56-59. Example MR guidance for pegged type anchorage is discussed belowwith reference to FIGS. 60-62. In each case, the anchorage may help inplacing a glenoid implant, such as a glenoid base plate for anatomicarthroplasty or a glenoid base plate and glenosphere for reversearthroplasty, onto the glenoid and fixing it in place.

FIG. 56 is a conceptual diagram illustrating a glenoid prosthesis withkeel type anchorage. As shown in FIG. 56, glenoid prosthesis 5600includes rear surface 5602 configured to engage a prepared surface ofglenoid 5102 (e.g., a reamed surface), and a keel anchor 5604 configuredto be inserted in a keel slot created in glenoid 5102 (e.g., keel slot5902 of FIG. 59).

FIGS. 57-59 are conceptual diagrams illustrating an MR system providingvirtual guidance for creating keel type anchorage positions in aglenoid, in accordance with one or more techniques of this disclosure.As shown in FIG. 57, MR system 212 may provide virtual guidance fordrilling additional holes in glenoid 5102. MR system 212 may provide thevirtual guidance for drilling the additional holes in any of a varietyof manners. As one example, MR system 212 may display virtual guidancesuch as virtual markers having specified shapes (e.g., axes, arrows,points, circles, X shapes, crosses, targets, etc.), sizes and/or colors,at the locations the additional holes are to be drilled. For instance,in the example of FIG. 57, MR system 212 may display virtual markers5700A and 5700B at the locations the additional holes are to be drilled.As another example, MR system 212 may display virtual axes at thelocations the additional holes are to be drilled to aid the surgeon inproperly aligning a drill bit to make the holes in the glenoid bone. Forinstance, MR system 212 may display (e.g., at the locations theadditional holes are to be drilled) a plurality of virtual drilling axeshaving respective parameters obtained from the virtual surgical plan,each respective virtual drilling axis of the plurality of virtualdrilling axes configured to guide drilling of a respective hole in theglenoid

MR system 212 may determine the locations of the additional holes basedon the virtual surgical plan. For instance, similar to virtual axis 5104of FIG. 51, MR system 212 may obtain, from the virtual surgical plan,the location(s) of the additional holes to be drilled on the virtualmodel of glenoid 5102. As such, by displaying virtual markers 5700A and5700B at the determined locations on the virtual model, MR system 212may display virtual markers 5700A and 5700B at the planned positions onglenoid 5102. As discussed above, the virtual surgical plan may bepatient specific in that the plan may be specifically developed for aparticular patient. As such, the planned positioned on glenoid 5102 atwhich MR system 212 displays virtual markers 5700A and 5700B may beconsidered patient-specific planned positions. Therefore, the locationsof the planned positions will vary from patient to patient according toindividual patient-specific surgical plans.

The surgeon may utilize a drill bit and a drill to create the additionalhole(s) at the location(s) indicated by MR system 212. For instance, asshown in FIG. 58, the surgeon may drill hole 5800A at the location ofvirtual marker 5700A and drill hole 5800B at the location of virtualmarker 5700B. The surgeon may use the same drill bit for each hole ormay use different drill bits for different holes.

MR system 212 may provide virtual guidance for the drilling in additionto or in place of the virtual markers, such as those described above,which indicate the locations the additional holes are to be drilled. Asone example, MR system 212 may provide targeting guidance to indicatewhether the drill is on a target axis. In this case, as an addition oralternative to the virtual markers, MR system 212 may display guide axesthat extend outward from the locations of each of the respective holesto be drilled. As another example, MR system 212 may display a mask withholes in the mask that correspond to the locations at which the holesare to be drilled. As another example, MR system 212 may display depthguidance to enable the surgeon to drill holes 5800A and 5800B to targetdepths (e.g., depth guidance discussed below with reference to FIGS.66-68).

MR system 212 may provide virtual guidance for working the holes into akeel slot that may accept keel anchor 5604 of glenoid prosthesis 5600.As an example, MR system 212 may display virtual outline 5802 aroundholes 5800A, 5500, and 5800B. For instance, MR system 212 may displayvirtual outline 5802 as approximately corresponding to a final outlineof the desired keel slot to be created.

The surgeon may utilize a tool to work holes 5800A, 5500, and 5800B intokeel slot 5902. As shown in FIG. 59, the surgeon may utilize keel punch5900 to work holes 5800A, 5500, and 5800B into keel slot 5902. Forinstance, the surgeon may impact keel punch 5900 into the area indicatedby virtual outline 5802. In this case, virtual outline 5802 defines ashape and dimension of the desired keel slot 5902, permitting thesurgeon to work the holes into a form that visually matches orapproximates the displayed virtual outline of the keel slot.

MR system 212 may provide additional or alternative virtual guidance forcreating keel slot 5902. As one example, MR system 212 may display depthguidance to enable the surgeon to impact keel punch 5900 to a targetdepth (e.g., depth guidance similar to the depth guidance discussedbelow with reference to FIGS. 66-68). As another example, MR system 212may provide targeting guidance to indicate whether keel punch 5900 is ona target axis (e.g., targeting guidance similar to the targetingguidance discussed below with reference to FIGS. 66-68). As anotherexample, MR system 212 may display a mask with a cutout for virtualoutline 5802.

FIG. 60 is a conceptual diagram illustrating a glenoid prosthesis withpegged type anchorage. As shown in FIG. 60, glenoid prosthesis 6000includes rear surface 6002 configured to engage a prepared surface ofglenoid 5102 (e.g., a reamed surface), a central peg anchor 6004configured to be inserted in a central hole created in glenoid 5102, andone or more peg anchors 6006A-6006C (collectively, “peg anchors 6006”)respectively configured to be inserted in additional holes created inglenoid 5102.

FIGS. 61 and 62 are conceptual diagrams illustrating an MR systemproviding virtual guidance for creating pegged type anchorage positionsin a glenoid, in accordance with one or more techniques of thisdisclosure. As shown in FIG. 61, MR system 212 may provide virtualguidance for drilling additional holes in glenoid 5102. MR system 212may provide the virtual guidance for drilling the additional holes inany of a variety of manners. As one example, MR system 212 may displayvirtual markers (e.g., axes, points, circles, X shapes, etc.) at thelocations the additional holes are to be drilled. For instance, in theexample of FIG. 61, MR system 212 may display virtual markers5700A-5700C at the locations the additional holes are to be drilled. Asanother example, MR system 212 may display virtual axes extending fromthe locations at which the additional holes are to be drilled. Asanother example, MR system 212 may display a mask (effectively aninverse of the virtual markers) that indicates where the holes are to bedrilled.

MR system 212 may determine the locations of the additional holes basedon the virtual surgical plan. For instance, similar to virtual axis 5104of FIG. 51, MR system 212 may obtain, from the virtual surgical plan,which may be patient-specific, the location(s) of the additional holesto be drilled on the virtual model of glenoid 5102. As such, bydisplaying virtual markers 5700A-5700C at the determined locations onthe virtual model, MR system 212 may display virtual markers 5700A-5700Cat the planned positions on glenoid 5102.

The surgeon may utilize a drill bit (or multiple drill bits) and a drillto create the additional hole(s) at the location(s) indicated by MRsystem 212. For instance, as shown in FIG. 62, the surgeon may drillhole 5800A at the location of virtual marker 5700A, drill hole 5800B atthe location of virtual marker 5700B, and drill hole 5800C at thelocation of virtual marker 5700C.

MR system 212 may provide virtual guidance for the drilling in additionto or in place of the virtual markers that indicate the locations theadditional holes are to be drilled. As one example, MR system 212 mayprovide targeting guidance to indicate whether the drill is on a targetaxis. As another example, MR system 212 may display depth guidance toenable the surgeon to drill holes 5800A-5800C to target depths.

It is noted that different implants may have different profiles, such asaugmented profiles. Additionally, as discussed herein, some implants maybe implanted with additional materials harvested from the patient, suchas bone grafts. In some of such examples, MR system 212 may providevirtual guidance for placement of the additional materials. Forinstance, MR system 212 may provide virtual guidance for attaching abone graft to an implant and guidance for attaching the graft/implantassembly to the patient.

In some examples, regardless of the anchorage type being used, thesurgeon may utilize a trial component to determine whether glenoid 5102has been properly prepared. The trial component may have a rear surfaceand anchors sized and positioned identical to the rear surface andanchors of the prosthesis to be implanted.

FIG. 149 is a flow diagram illustrating example techniques for MR aidedvalidation of anatomy preparation, in accordance with one or moretechniques of this disclosure. As discussed above, in some examples, atrial component may be used to determine whether a portion of apatient's anatomy has been properly prepared. For instance, a trialcomponent with surfaces matching the surfaces of a prosthetic to beimplanted may be placed in/around/against/etc. prepared anatomy. If thesurfaces of the trial component match up with the surfaces of theprepared anatomy, the surgeon may determine that the anatomy has beenproperly prepared. However, in some examples, it may be desirable todetermine whether anatomy has been properly prepared without requiringthe use of a trial component.

In accordance with one or more techniques of this disclosure, MR system212 may perform a virtual trialing to determine whether glenoid 5102 hasbeen properly prepared. For instance, MR system 212 may obtain one ormore dimensions of a glenoid of the scapula after the glenoid has beenprepared for attachment of the prosthetic (14902). As one example, oneor more sensors of MR system 212 (e.g., one or more depth cameras and/orone or more RGB cameras) may capture data of the prepared glenoid 5102.MR system 212 may determine, based on the captured data, one or moredimensions of the prepared glenoid 5102.

MR system 212 may obtain dimensions of a prosthetic to be implanted(14904). For instance, MR system 212 may obtain the dimensions of theprosthetic from a virtual surgical plan, a database of prostheticdimensions, or any other suitable source.

MR system 212 may compare the determined dimensions to the obtaineddimensions of the prosthetic to be implanted (14906) to determinewhether the anatomy has been properly prepared (14908). For instance, MRsystem 212 may determine that the anatomy has been properly preparedwhere a difference between the determined dimensions and the dimensionsof the prosthetic to be implanted is less than a threshold. Similarly,MR system 212 may determine that the anatomy has not been properlyprepared where a difference between the determined dimensions and thedimensions of the prosthetic to be implanted is greater than thethreshold. As one example, if a difference between the determineddimensions and the dimensions of the prosthetic to be implanted is lessthan a threshold, MR system 212 may determine that glenoid 5102 has beenproperly prepared (e.g., to receive the prosthetic). Similarly, if thedifference between the determined dimensions and the dimensions of theprosthetic to be implanted is greater than the threshold, MR system 212may determine that glenoid 5102 has not been properly prepared.

MR system 212 may output an indication of whether glenoid 5102 has beenproperly prepared to receive the implant, otherwise referred to as aprosthetic (14910/14912). As one example, MR system 212 may output agraphical indication that glenoid 5102 has been properly prepared toreceive the prosthetic. As another example, MR system 212 may output ahaptic or audible indication (e.g., via sensory devices 526) thatglenoid 5102 has been properly prepared to receive the prosthetic. Insituations where MR system 212 determines that glenoid 5102 has not beenproperly prepared, MR system 212 may provide virtual guidance foradditional work to be performed by the surgeon in order to properlyprepare glenoid 5102.

The above-described MR aided validation techniques may be used for anytype of anatomy, e.g., in any of a variety of joint repair surgicalprocedures. As one example, as described above, the MR aided validationtechniques may be used to determine whether a glenoid has been properlyprepared. As another example, the MR aided validation techniques may beused to determine whether a humerus has been properly prepared. Asanother example, the MR aided validation techniques may be used todetermine whether a tibia has been properly prepared. As anotherexample, the MR aided validation techniques may be used to determinewhether a talus has been properly prepared. As another example, the MRaided validation techniques may be used to determine whether a femur hasbeen properly prepared.

FIG. 63 is a conceptual diagram illustrating an MR system providingvirtual guidance for attaching an implant to a glenoid, in accordancewith one or more techniques of this disclosure. A tool may be used toattach the implant (e.g., a pegged implant, a keeled implant, or anyother type of implant) to glenoid 5102. For instance, the surgeon mayutilize impactor 6302 to insert prosthesis 6300 into the preparedglenoid 5102. In some examples, one or more adhesives (e.g., glue,cement, etc.) may be applied to prosthesis 6300 and/or glenoid 5102prior to impaction.

In some examples, one or more fasteners may be used to attach aprosthesis to glenoid 5102. For instance, as shown in FIGS. 64 and 65,screws 6400A-6400D (collectively, “screws 6400”) and central stem 6402may be used to attach prosthesis 6300 to glenoid 5102. These fastenersmay be used in addition to, or in place of, any anchorages included inthe prosthesis (e.g., pegs, keels, etc.).

MR system 212 may provide virtual guidance to facilitate theinstallation of the additional fasteners. For instance, as shown in FIG.65, MR system 212 may display virtual axes 6500A-6500D (collectively,“virtual axes 6500”), which may be referred to as “virtual screw axes,”to guide the surgeon in the installation of screws 6400 (e.g., intoglenoid 5102). In examples where screws 6400 are “self-tapping”, MRsystem 212 may display virtual guidance (e.g., virtual axes) to guideinsertion of screws 6400. For instance, MR system 212 may display avirtual screw axis having parameters (e.g., position, size, and/ororientation relative to the virtual model of the scapula) obtained fromthe virtual surgical plan that guides insertion of a screw into theglenoid. In examples where screws 6400 are not “self-tapping”, MR system212 may display virtual guidance (e.g., virtual axes) to guide drillingof pilot holes for screws 6400. For instance, MR system 212 may displaya virtual drilling axis having parameters (e.g., position, size, and/ororientation relative to the virtual model of the scapula) obtained fromthe virtual surgical plan that guides drilling of one or more holes(e.g., one or more pilot holes, and/or one or more clearance holes) inthe glenoid (e.g., for a screw of screws 6400).

To display the virtual guides for installation of the fasteners, MRsystem 212 may register a virtual model of the prosthesis to the actualobserved prosthesis. For instance, MR system 212 may obtain a virtualmodel of prosthesis 6300 from the virtual surgical plan and perform theregistration in a manner similar to the registration process describedabove with reference to FIGS. 17-20.

MR system 212 may obtain locations for each of the fasteners to beinstalled. For instance, MR system 212 may obtain, from the virtualsurgical plan, coordinates on the virtual model of the prosthesis andvector for each of the fasteners. In some examples, MR system 212 maydetermine that the coordinates for each fastener are the centroid of acorresponding hole in the prosthesis. For instance, MR system 212 maydetermine that the coordinates for screw 6400A are the centroid of hole6502.

The surgeon may install the fasteners using the displayed virtualguidance. For instance, the surgeon may use a screwdriver or otherinstrument to install screws 6400.

MR system 212 may display virtual guidance to assist in the fastenerattachment. As one example MR system 212 may provide depth guidance. Forinstance, MR system 212 may display depth guidance to enable the surgeonto install each of screws 6400 to a target depth. As another example, MRsystem 212 may provide targeting guidance. For instance, MR system 212may display an indication of whether each of screws 6400 is beinginstalled on a prescribed axis. As another example, MR system 212 mayprovide guidance on an order in which to tighten screws 6400. Forinstance, MR system 212 may display a virtual marker on a particularscrew of screws 6400 that is to be tightened.

As discussed above, MR system 212 may provide a wide variety of virtualguidance. Example of virtual guidance that may be provided by MR system212 include, but are not limited to, targeting guidance and depthguidance. MR system 212 may provide targeting guidance to assist asurgeon in performing work (e.g., drilling a hole, reaming, installing ascrew, etc.) along a particular axis. MR system 212 may provide depthguidance to assist a surgeon in performing work (e.g., drilling a hole,reaming, installing a screw, etc.) to a desired depth.

FIG. 66 is a conceptual diagram of virtual guidance that may be providedby an MR system, in accordance with one or more techniques of thisdisclosure. As shown in FIG. 66, a surgeon may view a portion of scapula5100 through visualization device 213.

As discussed above, in some examples, the surgeon may utilize one ormore tools to perform work on portion of a patient's anatomy (e.g.,scapula 5100, humerus 3200, etc.). For instance, the surgeon may utilizea drill, such as drill 6600, to install guide 5200, operate reaming tool5300, operate drill bit 5400, and/or install screws 6400. However, asshown in FIG. 66, drill 6600 may obstruct at least some of the portionof scapula 5100 on which the surgeon is performing the work.

In accordance with one or more techniques of this disclosure, MR system212 may utilize visualization device 213 to provide a surgeon withgraphical targeting guidance 6602 and graphical depth guidance 6604 forwork the surgeon is performing on the portion of scapula 5100 (orhumerus 3200). For instance, to display graphical targeting guidance6602, MR system 212 may determine a current orientation of the toolbeing used (e.g., drill 6600 or a bit/instrument attached thereto) and atarget orientation (e.g., a target vector obtained via a virtualsurgical plan). MR system 212 may determine the current orientation ofthe tool being used using any number of techniques. As one example, MRsystem 212 may determine the current orientation of the tool being usedbased on orientation data received from one or more sensors (e.g.,accelerometers, gyroscopes, etc.) attached to the tool. As anotherexample, MR system 212 may determine the current orientation of the toolbeing used based on data captured by one or more sensors ofvisualization device 213 (e.g., optical cameras 530, depth cameras 532,etc.).

In some examples, MR system 212 may display graphical targeting guidance6602 as a graphical representation of one or more viewpoints (inaddition to the viewpoint of the surgeon). The graphical representationsmay be referred to as synthesized views. For example, as shown in FIG.66, MR system 212 may display a graphical representation of a side viewfrom the viewpoint indicated by arrow 6606 and a graphicalrepresentation of a top view from the viewpoint indicated by arrow 6606.The side view may be a left side view or a right-side view. Similarly,the top view may be a bottom view. In some examples, the display oftop/bottom/left/right views may be selectable by the surgeon.Additionally or alternatively, MR system 212 may automatically selectwhich views to display (e.g., select default views) based on whether thesurgical procedure is being performed on a left or right side of thepatient.

In FIG. 66 the graphical representations of the additional viewpointsshow a relative angular difference between a current orientation of thetool being used (e.g., drill 6600 or a bit/instrument attached thereto)and a target orientation (e.g., a target vector). Other graphicalrepresentations that may be displayed by MR system 212 include, but arenot limited to, reticles, numerical values (e.g., a numerical value ofthe relative angular difference), symbolic representations (e.g.,checkmarks/Xes, colored shading, etc.), or any other graphicalrepresentation of a relationship between a current orientation of thetool being used and a target vector extending to a target position onthe bone (e.g., a glenoid of a scapula, a humerus, or any other bone).It is understood that MR system 212 may display the additionalviewpoints in addition to or in place of other virtual guidance. Forinstance, MR system 212 may simultaneously display the additional viewpoints and a virtual axis/cutting surface.

For instance, to display graphical depth guidance 6604, MR system 212may determine a current depth of the tool being used (e.g., drill 6600or a bit/instrument attached thereto) and a target depth (e.g., a targetdepth obtained via a virtual surgical plan). MR system 212 may determinethe current depth of the tool being used using any number of techniques.As one example, MR system 212 may determine the current depth of thetool being used based on orientation data received from one or moresensors (e.g., accelerometers, gyroscopes, etc.) attached to the tool.As another example, MR system 212 may determine the current depth of thetool being used based on data captured by one or more sensors ofvisualization device 213 (e.g., optical cameras 530, depth cameras 532,etc.). As another example, MR system 212 may determine the current depthof the tool as described herein with reference to FIGS. 73-79.

MR system 212 may display graphical depth guidance 6604 as a graphicalrepresentation of a relationship between the current depth and thetarget depth. In FIG. 66 the graphical representation of therelationship between the current depth and the target depth is shown asan arrow that advances from a starting depth (illustrated as a whitecircle in this example) and the target depth (illustrated as a blackcircle in this example). Other graphical representations that may bedisplayed by MR system 212 include, but are not limited to, numericalvalues (e.g., a numerical value of the remaining depth; a numericalvalue of the current depth and a numerical value of the target depth;etc.), symbolic representations (e.g., a checkmark when the target depthhas been achieved, colored shading, etc.), or any other graphicalrepresentation of a relationship between the current depth and thetarget depth.

As discussed herein, visualization device 213 of MR system 212 maydisplay the virtual guidance on a lens through-which the surgeon isviewing the patient's anatomy. As such, in some examples, the virtualguidance may be regarded as mixed reality (MR) guidance.

MR system 212 may output guidance in other forms in addition to or inplace of the graphical guidance (targeting and/or depth). For instance,MR system 212 may output audio and/or haptic guidance for one or both oftargeting and depth. As one example, MR system 212 may output hapticdepth guidance by causing one or both of a tool currently being usedand/or visualization device 213 to vibrate when the target depth isreached. As another example, MR system 212 may output haptic targetingguidance by causing one or both of a tool currently being used and/orvisualization device 213 to vibrate when the orientation of the tooldrifts more than a threshold amount from the target orientation.

As another example, MR system 212 may output audio depth guidance bycausing visualization device 213 to output an audible representation ofa difference between the current depth and the target depth. Forinstance, visualization device 213 may output a tone that increases involume as the target depth is approached and changes frequency when thetarget depth is reached. As another example, MR system 212 may outputaudio targeting guidance by causing visualization device 213 to outputaudio to indicate when the orientation of the tool drifts more than athreshold amount from the target orientation.

For instance, visualization device 213 may output a tone in a left earof the surgeon in response to determining that the tool is drifting leftof the target orientation and a tone in a right ear of the surgeon inresponse to determining that the tool is drifting right of the targetorientation. Similarly, visualization device 213 may output a toneperceived by the surgeon as being above the surgeon in response todetermining that the tool is drifting up from of the target orientationand a tone perceived by the surgeon as being below the surgeon inresponse to determining that the tool is drifting down from the targetorientation

In some examples, visualization device 213 (e.g., via one or more ofsensory devices 526) may output the audio targeting guidance usingthree-dimensional audio. For instance, visualization device 213 mayoutput a tone to be perceived as being in front of/to the left of/to theright of/behind/above/below the surgeon in response to determining thatthe tool is drifting in a corresponding direction.

While illustrated as being from certain view angles and orientations, itis noted that the virtual guidance described herein may be displayedfrom any view angle or orientation. For instance, the relative positionof a virtual guide (e.g., axis, point, surface, etc.) to an observedanatomical object may be maintained even as a user of visualizationdevice 213 moves their head and/or moves around the patient's anatomy.

As discussed above, MR system 212 may register virtual models of apatient's anatomy with corresponding observed anatomy. As one example,MR system 212 may register a virtual model of a patient's glenoid withthe patient's actual glenoid as observed by one or more sensors ofvisualization device 213. As another example, MR system 212 may registera virtual model of a patient's humerus with the patient's actual humerusas observed by one or more sensors of visualization device 213.

In some examples, an initial registration may be sufficient to supportMR system 212 providing virtual guidance for all work steps performed ona particular piece of anatomy. For instance, an initial registration ofthe virtual model of a patient's glenoid with the patient's actualglenoid as observed by one or more sensors of visualization device 213may be sufficient to enable MR system 212 to provide virtual guidancefor all work steps performed on the glenoid (e.g., installation of aguide pin, reaming, creating anchorage points, etc.).

However, in some examples, MR system 212 may perform additionalregistrations. For instance, after performing a particular work step ona piece of anatomy, MR system 212 may re-register a virtual model of thepiece of anatomy that takes into account the work step performed (e.g.,an additional virtual model that incorporates planned work). As oneexample, after a surgeon reams a patient's glenoid, MR system 212 mayobtain a subsequent virtual model of the glenoid that includes theplanned reaming and register the subsequent virtual model to theobserved reamed glenoid in a manner similar to the registration processdescribed above with reference to FIGS. 17-20.

In some examples, MR system 212 may be configured to provideinstructions for how the surgeon is to perform steps of the workflow.For instance, in response to user input (e.g., voice commands, gestures,etc.), MR system 212 may display an animation, video, or text todescribe how a particular step or steps are to be performed. Theinstructions may be general or may be patient specific.

As discussed above, MR system 212 may provide virtual guidance forperforming work on a patient. For instance, visualization device 213 maydisplay virtual guidance to a surgeon using visualization device 213.Additionally, as discussed herein, other individuals may usevisualization devices. MR system 212 may provide virtual guidance tomultiple individuals using respective visualization devices. The virtualguidance may be same for all individuals or may different. For instance,MR system 212 may provide different virtual guidance to differentindividuals that is tailored to the individuals' roles in the operatingroom.

For purposes of illustration several figures show more of the patient'sanatomy than would actually be visible during surgery (e.g., as portionswould be obscured by tissue). For instance, FIG. 34 illustrates more ofhumerus 3200 than would actually be visible during surgery and FIG. 51illustrates more of scapula 5100 than would actually be visible duringsurgery.

As discussed above, in some examples, MR system 212 may display avirtual model of a portion of a patient's anatomy overlaid on anobserved portion of the patient's anatomy. For instance, during aregistration procedure, visualization device 213 may display a 3Dvirtual model of a patient's glenoid overlaid on the patient's actualglenoid. The 3D virtual model may appear, for example, to be within areal-world environment with the actual glenoid. In some examples, MRsystem 212 may only display the virtual model during the registrationprocess. In other examples, MR system 212 may display the virtual modelduring other portions of the surgical procedure. For instance, MR system212 may display a virtual model of a portion of a patient's anatomyoverlaid on an actual portion of the patient's anatomy while the surgeonperforms work (e.g., cutting, drilling, reaming, prosthesis attachment,trialing, etc.) on the actual portion of the patient's anatomy.

In some examples, MR system 212 may display the virtual model such thatthe virtual model obscures a corresponding portion of the patient'sanatomy. In some examples, MR system 212 may display the virtual modelsuch that the virtual model does not completely obscure a correspondingportion of the patient's anatomy (e.g., display at least some of thevirtual model overlaid on the corresponding portion of the patient'sanatomy). As one example, MR system 212 may display the virtual modelsuch that a region of interest on the actual portion of the patient'sanatomy (e.g., a location on the patient's anatomy at which work is tobe performed) is not obscured (at least not totally obscured) by thevirtual model. For instance, MR system 212 may display the virtual modelsuch that a location on the anatomy indicated by a virtual guide (e.g.,corresponding to the region of interest on the actual portion of thepatient's anatomy) is not completely obscured by the displayed virtualmodel. For example, MR system 212 may display a virtual model of aglenoid with a “hole” or missing region surrounding the area at whichthe guide pin is to be installed. The “hole” or missing region may bereferred to as a virtual hole in the virtual model. As another example,MR system 212 may display an outline of the virtual model.

FIGS. 69 and 70 are conceptual diagrams illustrating example virtualmodels with missing regions surrounding areas of interest. Similar toFIG. 51, FIGS. 69 and 70 are other examples of virtual guidance toassist a surgeon in installing a guide pin. As shown in FIGS. 69 and 70,MR system 212 may display virtual bone model 1008 as including virtualhole 6900 surrounding a region that includes a virtual guide such asvirtual axis 5104 (i.e., a region corresponding to a location on theanatomy indicated by a virtual guide). As can be seen, while at leastsome of virtual bone model 1008 is still displayed, a portion of glenoid5102 is unobscured by virtual bone model 1008 due to the inclusion ofvirtual hole 6900.

In some examples, the virtual hole may have a hard boundary. In someexamples, MR system 212 may display the virtual model as fading (e.g.,in opacity) as the location on the anatomy indicated by the virtualguide is approached. For instance, MR system 212 may display a perimeterof the virtual model with a particular opacity and reduce the opacity(e.g., fade the virtual model out) in the direction of the location onthe anatomy.

FIG. 71 is a conceptual diagram illustrating an example virtual modelwith a missing region surrounding an area of interest. Similar to FIG.57, FIG. 71 is another example of virtual guidance to assist a surgeoncreating anchorage points. As shown in FIG. 71, MR system 212 maydisplay virtual bone model 1008 as a perimeter around glenoid 5102. Ascan be seen, while at least some of virtual bone model 1008 is stilldisplayed, a portion of glenoid 5102 is unobscured by virtual bone model1008.

Presenting a virtual hole in the virtual model may provide the surgeonwith a better view of an actual, real anatomical surface, such as a bonesurface, promoting visibility of the surface as the surgeon performsoperations to prepare the surface to receive a prosthetic implant oroperations for placement of the prosthetic implant on the surface.Maintaining at least some display of the virtual model while work isbeing performed may provide one or more advantages. For instance, if atleast a portion of the virtual model (e.g., a perimeter, an outline, avirtual model with a hole, etc.) is displayed while work is beingperformed with the assistance of virtual guidance, the surgeon can beconfident that MR system 212 is displaying the virtual guidance at theproper location(s). In particular, as the surgeon would be able to seethat the displayed portion of the virtual model is properly aligned withthe observed anatomy and the virtual guidance is displayed based on theposition of the virtual model, the surgeon would be confident that MRsystem 212 is displaying the virtual guidance at the proper location(s).

As discussed above, in some examples, the surgeon may utilize one ormore tools to perform work on portion of a patient's anatomy (e.g.,scapula 5100, humerus 3200, etc.). For instance, the surgeon may utilizea drill, such as drill 6600, to install guide 5200, operate reaming tool5300, operate drill bit 5400, and/or install screws 6400. Such tools maybe powered and controllable, e.g., by a manual button, trigger, orswitch. As also discussed above, in some examples, MR system 212 mayprovide virtual guidance to assist in the performance of the work. Forinstance, MR system 212 may provide targeting guidance, depth guidance,and the like.

In accordance with one or more techniques of this disclosure, a MRsystem may be configured to positively control operation of one or moretools used to perform work. For instance, MR system 212 may selectivelyadjust operation of a drill, such as drill 6600, based on whether thedrill is accurately positioned on a virtual axis. Adjustment ofoperation may include powering the tool on, powering the tool off,enabling the tool to be manually powered on by the surgeon, disablingthe tool from being manually powered on by the surgeon, and/orcontrolling a speed, torque or force of the tool, such as a rotationalspeed, torque or force. In this way, MR system 212 may achieveclosed-loop control over one or more tools. In some examples,closed-loop control may be a function of the monitored positioning ofthe tool or a portion of the tool relative to prescribed axes,positions, angles or the like, which, in some examples, also may beillustrated visually to the surgeon by display of virtual guidance.

FIG. 72 is a flow diagram illustrating example techniques for closedloop tool control in surgical procedures, in accordance with one or moretechniques of this disclosure. In operation, MR system 212 may obtain,from a virtual surgical plan, a target value of a parameter of amodification to be made to a portion of a patient's anatomy with a tool(7200). Examples of parameters include, locations at which holes are tobe drilled, depths (e.g., drilling depths, reaming depths, etc.),locations of cutting surfaces, etc. As one specific example, MR system212 may obtain a location on a glenoid at which a guide pin is to beinstalled (e.g., the location on glenoid 5102 at which MR system 212 isdescribed as displaying virtual axis 5104 in FIG. 51). As anotherspecific example, MR system 212 may determine a depth to ream a glenoid.

MR system 212 may obtain a current value of the parameter (7202). As oneexample, MR system 212 may determine a current location/depth of thetool being used based on orientation data received from one or moresensors (e.g., accelerometers, gyroscopes, etc.) attached to the tool.For instance, MR system 212 may determine a current orientation of thetool based on data measured by an accelerometer attached to the tool. Asanother example, MR system 212 may determine the current location/depthof the tool being used based on data captured by one or more sensors ofvisualization device 213 (e.g., optical cameras 530, depth cameras 532,etc.). For instance, MR system 212 may register a virtual model of thetool to the observed tool. As another example, MR system 212 maydetermine the current location/depth of the tool as described hereinwith reference to FIGS. 73-79.

MR system 212 may compare the current value of the parameter and thetarget value of the parameter (7204). As one example, MR system 212 maydetermine a difference between a current orientation/axis of the tooland a target axis. In some examples, the target axis may be illustratedby a virtual guidance axis displayed by MR system 212 for viewing by thesurgeon, e.g., to aid the surgeon in aligning the tool (e.g., such as adrill bit reaming shaft) with an axis prescribed by the surgical plan.Similarly, other virtual guidance such as virtual placement markers,angle indications or depth indications and/or other virtual guidance maybe displayed as virtual guidance during detection and modification of aparameter of the tool. In other examples, the parameter or parameters ofthe tool may be controlled, e.g., by MR system 212, and the parametermonitored without displaying a virtual guidance axis, virtual placementmarkers, virtual angle indications, virtual depth indications, and/orother virtual guidance. For instance, MR system 212 may determine thatthe current orientation/axis of the tool is 5 degrees off from a targetaxis (e.g., MR system 212 may determine a difference between the currentorientation of the tool (as determined based on data measured by anaccelerometer attached to the tool) and a prescribed axis is 5 degrees).As another example, MR system 212 may determine a difference between acurrent depth of the tool and a target depth. For instance, where thecurrent depth is 3 mm, e.g., from an initial drilling surface plane, andthe target depth is 7 mm, MR system 212 may determine that the currentdepth is 4 mm short of the target depth.

MR system 212 may automatically and selectively adjust a state of thetool based on the comparison (7206). As one example, MR system 212 maygate operation of the tool based on the comparison. For instance, MRsystem 212 may allow operation of the tool where the difference betweenthe target value and the current value is less than a threshold. Forexample, MR system 212 may activate the tool or enable power to permit asurgeon to manually activate the tool. Similarly, MR system 212 mayprevent operation of the tool where the difference between the targetvalue and the current value is greater than a threshold. For example, MRsystem 212 may deactivate the tool or disable power so that the surgeonis not permitted to manually activate the tool. As another example, MRsystem 212 may throttle speed, torque, force or another operatingparameter or otherwise limit operation of the tool based on thecomparison. For instance, MR system 212 may allow operation of the toolat full power where the difference between the target value and thecurrent value is less than a threshold. Similarly, MR system 212 mayreduce the operating power (e.g., rotational speed, torque output, etc.)of the tool where the difference between the target value and thecurrent value is greater than a threshold. For instance, MR system 212may reduce the operating power by some function of the differencebetween the target value and the current value.

While MR system 212 is described as automatically and selectivelyadjusting the state of the tool, actual activation/deactivation of thetool may be manually performed by the surgeon. For instance, where MRsystem 212 is allowing operation of the tool, the tool may not actuallyactivate until the surgeon pulls a trigger or actuates a button or othercontrol input of the tool, e.g., a trigger or button of a drill motor ofthe tool. Additionally, in some examples, MR system 212 and/or the toolmay include an override that enables the surgeon to activate the toolregardless of whether MR system 212 is currently allowing operation ofthe tool.

MR system 212 and the tool may communicate via any suitable manner. Bothwired and wireless communication links are contemplated. For instance,MR system 212 and the tool may communicate over a Bluetooth link, aWi-Fi link (e.g., according to any of the IEEE 802.11 standards), awired serial connection (e.g., RS-232 or USB), 3G, 4G or 5G, or anyother suitable communication link.

MR system 212 may enforce the positive control over the tool via thecommunication link or any other technique. As one example, MR system 212may output, via the communication link, data to the tool indicatingwhether operation of the tool is being gated or throttled. As anotherexample, MR system 212 may control a power supply of the tool such thatMR system 212 may prevent the tool from receiving power (or adjust theamount of power received) when operation of the tool is being gated orthrottled. For instance, the tool may be powered via a “smart plug” andMR system 212 may control activation of the smart plug when operation ofthe tool is being gated or throttled. In addition, the activated,deactivated or otherwise adjusted status of the tool may be communicatedto the surgeon, e.g., audibly or visually via augmented reality (AR) orMR content displayed by MR system 212.

The tool may be any type of tool usable by a surgeon. Examples of toolsinclude, but are not limited to, drills, saws, lasers, and any othertype of tool used during surgery. Such tools may be manually driven bythe surgeon or powered by a motor, such as a rotating or reciprocatingmotor or a vibrating motor, such as a piezoelectric generator. The toolmay be considered to be manually guidable in that theposition/orientation/location of the tool is controlled by the surgeonby moving and/or rotating the tool. As such, in some examples, the toolmay be considered to be a hand-held tool, such as a hand-held drill,e.g., having a drill motor and a chuck or other interface to receive ashaft carrying a working bit, such as a drill bit, reaming element, orother working element.

The positive control of the tool may be used in conjunction with anyother techniques of this disclosure. For instance, MR system 212 mayboth provide virtual graphical guidance and positive tool control asdescribed above. As one specific example, MR system 212 may provide thevirtual graphical guidance discussed above with reference to FIG. 66 atthe same time MR system 212 is performing closed-loop tool control.

FIG. 67 is a conceptual diagram of an example view 6700 that may beprovided by an MR system and that provides a secondary view window, inaccordance with one or more techniques of this disclosure. The exampleof FIG. 67 shows what a surgeon may see while using an MR visualizationdevice (e.g., visualization device 213) during an orthopedic shouldersurgery. Particularly, in the example of FIG. 67, the surgeon may viewan exposed portion of scapula 5100 and an area of tissue 6702 thatsurrounds the exposed portion of scapula 5100.

As discussed above with respect to FIG. 66, the surgeon may use one ormore tools to perform work on portion of a patient's anatomy (e.g.,scapula 5100, humerus 3200, etc.). For instance, the surgeon may use adrill, such as drill 6600, to install guide 5200, operate reaming tool5300, operate drill bit 5400, and/or install screws 6400. However, asshown in FIG. 67, drill 6600 may obstruct at least some of the portionof scapula 5100 on which the surgeon is performing the work.Furthermore, it may be challenging for the surgeon to assess how deeplya tool, such as a bit of drill 6600, has penetrated a tissue or a bone.This may be especially challenging when the surgeon is looking down thelength of a drill, such as drill 6600.

Hence, in accordance with one or more techniques of this disclosure,visualization device 213 of MR system 212 may generate a MRvisualization that includes a secondary view window 6704, which may be asub-window overlaid or otherwise composed with any contents, such asother virtual guidance, of a main window. Secondary view window 6704,along with other virtual guidance (e.g., virtual markers, depthguidance, etc.) may appear along with physical, real-world objects inthe surgeon's field of view. Thus, in the example of FIG. 67, thesurgeon may see secondary view window 6704 along with the exposedportion of scapula 5100, tissue 6702, and drill 6600, as well as anyvirtual guidance such as a virtual axis or virtual entry point. In someexamples, the surgeon or other user may resize or reposition secondaryview window 6704.

Secondary view window 6704 contains images representing a differentperspective on a surgical site. For instance, in the example of FIG. 67,the surgical site is a patient's glenoid and the surgeon is drilling ahole in the exposed glenoid portion of the patient's scapula 5100.Furthermore, in the example of FIG. 67, the surgeon's perspective issubstantially down an axis of drill 6600. Relative to the patient, thesurgeon's perspective in FIG. 67 is in a frontal axis of the patient.Accordingly, in the example of FIG. 67, secondary view window 6704contains images representing the glenoid portion of the patient'sscapula from a perspective other than down the axis of drill 6600. Thatis, in the example of FIG. 67, the images in secondary view window 6704are not in any frontal axis of the patient. Rather, in the example ofFIG. 67, the images presented in secondary view window 6704 are from aperspective 90° rotated from the perspective of the surgeon. Forinstance, relative to the patient, the images presented in secondaryview window 6704 may be in a sagittal axis.

The surgeon may use secondary view window 6704 to check the depth towhich the tool has penetrated. For instance, in the example of FIG. 67,the surgeon may use secondary view window 6704 to determine how far abit of drill 6600 has penetrated scapula 5100.

The images presented in secondary view window 6704 may be generated invarious ways. For instance, in one example, the images presented insecondary view window 6704 may be captured by a video camera. In somesuch examples, the video camera may be worn or held by a person otherthan the surgeon, such as a nurse. For instance, the video camera may bemounted on a visualization device worn by a nurse (e.g., a visualizationdevice of MR system 1800B (FIG. 18). In some examples, the video cameramay be mounted on a fixed wall, mechanical arm, robot, or other physicalobject.

In other examples, the images presented in secondary view window 6704may comprise or consist of virtual objects. For instance, the imagespresented in secondary view window 6704 may include a virtual3-dimensional model of the patient's anatomy. Additionally, the imagespresented in secondary view window 6704 may include a virtual3-dimensional model of a tool being used by the surgeon. Thus, in theexample of FIG. 67, secondary view window 6704 may include a virtual3-dimensional model of the patient's scapula 5100 and a virtual3-dimensional model of drill 6600. The virtual 3-dimensional model ofthe patient's anatomy may be the same as that used during preoperativeplanning of the surgery. In addition, in some examples, secondary viewwindow 6704 may include virtual guidance such as a virtual reaming ordrilling axis, a virtual cutting plan, a virtual entry point or thelike.

In examples where the images presented in secondary view window 6704comprise or consist of virtual objects, the patient's anatomy may beregistered with a corresponding virtual model of the patient's anatomy,as described elsewhere in this disclosure. For instance, the patient'sglenoid may be registered to a virtual model of the patient's glenoid.Thus, a computing system (e.g., MR system 212 (FIG. 2) may be able todetermine the position and orientation of the patient's anatomy in a3-dimensional space. Furthermore, the computing system may receiveinformation from one or more sensors (e.g., cameras, motion sensors,etc.) that enable the computing system to determine a location of a tool(e.g., drill 6600) in the same 3-dimensional space. One or more markerson the tool may assist the computing system in identifying the locationof the tool. Accordingly, the computing system may determine theposition of the tool relative the patient's anatomy. The computingsystem may generate the images of secondary view window 6704 based onthe relative positions of the patient's anatomy and the tool. Thus, inthe example of FIG. 67, the computing system may generate a MRvisualization in secondary view window 6704 that shows the relativepositions of the virtual 3-dimensional models of the patient's scapula5100 and a bit of drill 6600.

Presenting virtual 3-dimensional models of the patient's anatomy and atool used by the surgeon may address a certain set of challenges. Forinstance, in examples where a nurse holds or wears a camera that feedsimages into secondary view window 6704, the nurse's natural movementsmay create camera shake that may be distracting to the surgeon. Tocompensate for camera shake, a computing system may need to apply imagestabilization, which may be computationally expensive, potentiallyresulting in battery drain, and may result in a reduced field of view.Furthermore, virtual 3-dimensional models in secondary view window 6704do not suffer from camera shake in this way, which may conservecomputation resources otherwise expended on image stabilizing, as wellas potentially increased field of view and reduced surgeon distraction.

Another potential advantage of using virtual 3-dimensional models may bethat unneeded background information may be omitted from secondary viewwindow 6704. For instance, in the example of FIG. 67, tissue 6702 may beomitted from the images presented in secondary view window 6704.Omitting unneeded background information may further reduce visualdistraction for the surgeon. Furthermore, the surgeon may be able torotate or otherwise change the perspective of the virtual 3-dimensionalmodels shown in secondary view window 6704 to angles that may beimpractical for a human nurse to obtain with a handheld or head-worncamera. Accordingly, fixed position video cameras or mechanical-armmounted cameras may need to be used to achieve the perspective that thesurgeon may want. The use of virtual 3-dimensional models may eliminatethe need for expending hospital resources on such cameras and mountingsystems.

In some examples, MR system 212 may display graphical depth guidance bydisplaying an illustration of a location of a portion of the anatomy anda location of a tool relative to the portion of the anatomy. Forexample, as illustrated in FIG. 68, MR system 212 may display asecondary view window 6704 that includes image 6802, which depicts alocation of drill 6600 relative to scapula 5100. As shown in FIG. 68,image 6802 may further depict current depth 6804 and target depth 6806.As discussed above, the depth guidance may include a numerical value ofa difference between current depth 6804 and target depth 6806 (i.e., “3mm remaining”).

MR system 212 may render the illustration of the location of the portionof the anatomy and the location of the tool relative to the portion ofthe anatomy may from any view (e.g., top view, left side view, rightside view, above view, below view, etc.). In the example of FIG. 68, MRsystem 212 renders the image from the side view indicated by arrow 6800.

As described elsewhere in this disclosure, a computing system (e.g.,virtual planning system 102) may generate an information model of asurgery. The information model of the surgery describes a surgicalworkflow that comprises a series of steps that a surgeon would performin order to complete a surgery. For example, the surgical workflow maycomprise the series of steps shown in the example of FIG. 19.Furthermore, as described elsewhere in this disclosure, a computingsystem (e.g., intraoperative guidance system 108 (FIG. 1), computingsystems 11706 (FIG. 117), etc.) may mark steps of the surgical workflowas complete as the surgeon progresses through the surgical workflow.

In accordance with a technique of this disclosure, MR system 212 mayautomatically display secondary view window 6704 at appropriate steps ofthe surgical workflow and hide secondary view window 6704 at other stepsof the surgical workflow. For example, MR system 212 may show secondaryview window 6704 during steps of the surgical workflow in which thesurgeon uses a drill and hide secondary view window 6704 otherwise.Furthermore, in some examples, default perspectives of secondary viewwindow 6704 when showing virtual 3-dimensional models may be tied tosteps in the surgical workflow. For instance, in one example, secondaryview window 6704 may have a default perspective along a frontal axis ofthe patient in one step of the surgical workflow and may have a defaultperspective along a longitudinal axis of the patient in another step ofthe surgical workflow. The default perspective may be a perspective ofsecondary view window 6704 presented before a user modifies theperspective of secondary view window 6704.

Another aspect of this disclosure is directed to a mixed reality(MR)-based technique for tracking and presenting depth of a tooling bitin a medical procedure. The techniques may be particularly useful fororthopedic medical procedures, such as shoulder surgeries, anklesurgeries, knee surgeries, hip surgeries, or any joint repair surgicalprocedure or augmentation. Although the techniques may be useful in awide variety of orthopedic procedures, they may be especially useful inboth anatomical and reverse-anatomical shoulder reconstructionsurgeries. Indeed, the techniques may be helpful for reversedarthroplasty, augmented reverse arthroplasty, standard total shoulderarthroplasty, augmented total shoulder arthroplasty, hemisphericalshould surgery, or other types of shoulder surgery. Even morespecifically, the techniques may be especially useful in mixed reality(MR)-based techniques for glenoid reaming, e.g., reaming of glenoid boneto condition a bone surface to receive an implant, in an anatomicalshoulder reconstruction surgery.

In general, the tooling bit that is being tracked for depth-tracking maycomprise any type of medical tool, including a drill bit, a reamingelement, a grinding element, or any element that is configured to rotateon a rotating shaft. The techniques may be used for drilling, reaming orgrinding to a desired depth, e.g., relative to a fixed or knownlocation, such as a starting point of the tooling bit at the start ofthe drilling, reaming or grinding process. In some examples, thestarting point is defined by a virtual plane, which may be selected bythe user to initialize the drilling, reaming or grinding process. Afterdefining the starting point, downward movement of the depth aid elementalong the axis can be used to track downward depth of the tooling bit.

For anatomical shoulder reconstruction surgeries, for example, thetechniques and tools described herein may be well-suited for trackingdepth of a reaming tool when performing reaming on a glenoid bone of apatient. In some examples, mixed reality (MR) devices (e.g., mixedreality headset), such as visualization device 213 of MR system 212, maybe used to implement one or more depth cameras according to thisdisclosure. An MR device such as visualization device 213 may be anexample of an MR system 212. For instance, an MR device may be an MRsystem that is enclosed within a housing. The MR device, equipped withone or more depth cameras, may be configured to detect and track a depthof point of a tooling bit relative to a target depth. In some examples,the MR device is able to track displacement relative to a startingpoint, which may be determined by a registration process. For example,after drilling and inserting a guide pin, and upon placing a reamingelement at a reaming location of the guide pin, a registration processmay be performed on a depth tracking element, and thereafter, depthtracking of the depth tracking element may be used as a proxy for depthtracking of the reaming element.

Depth tracking of the tooling bit may be based upon depth tracking ofanother element that is positioned a fixed and known distance from thetooling bit along a rotational axis associated with the tooling bit. Insome examples, the techniques use mixed reality to track the depth ofsomething that moves co-axially with the tooling bit. For example, thetechniques may use a depth aid element located at a fixed and knownlocation relative to the tooling bit. Or alternatively, rather thanadding a depth aid element, the techniques may track a medical drillhousing, or another element of the medical device. In yet anotherexample, the techniques may track a backside of the tooling bit, whichmay be the back side of a reaming element.

Mixed reality may be used for the tracking, and a registration processmay be performed to register a virtual element to the medical device.For example, the virtual element may be registered to a depth aidelement of the device, or possibly to a medical device housing of asurgical drill, or possibly to the backside of the tooling bit. Thevirtual element may correspond to a shape of a portion of the device ora shape of a depth aid element, or in some cases, the virtual elementmay simply comprise depth plane that is orthogonal to an axis of thetooling bit and positioned a fixed and known distance from the toolingbit. This disclosure contemplates depth tracking with depth cameras. Insome cases, the depth cameras are part of a mixed reality system such asMR system 212, but in other cases, the depth cameras may operate outsideof a mixed reality environment.

FIG. 73 is a conceptual side view of a portion of a medical device 7300and depth cameras 7314 consistent with an example of this disclosure. Insome examples, drill 6600 (FIG. 66, FIG. 67) is an instance of medicaldevice 7300. Medical device 7300 comprises a rotatable shaft 7302 and atooling bit 7304 located on a distal end of rotatable shaft 7302. Inthis example, tooling bit 7304 may comprise a drilling element, e.g.,designed for drilling into bone of a patient, or a reaming element,e.g., designed for reaming bone of a patient. The opposing proximal end7308 of rotating shaft 7302 may be coupled to a medical drill thatrotates shaft 7302 during the medical procedure. The medical drill maycomprise a powered drill that is manually controlled by user actuationof a trigger or other user input or automatically controlled by acontroller, e.g., in response to control signals such as control signalsgenerated responsive to detected depth of the tooling bit. The medicaldrill may include a surgical motor that is coupled to shaft 7302 torotate the shaft.

As shown in FIG. 73, medical device 7300 includes a depth aid element7312 positioned at a fixed location relative to an axis of rotatableshaft 7302. Since depth aid element 7312 is positioned at a fixedlocation relative to tooling bit 7304 along an axis of rotatable shaft7302, depth aid element 7312 moves co-linearly with tooling bit 7304along the axis. In other words, depth aid element 7312 is positioned aknown, fixed distance from tooling bit 7304 along rotatable shaft 7302.Therefore, tracking of the location of depth aid element 7312 along theaxis can be used as a proxy for tracking of the location of tooling bit7304. Accordingly, upon placing tooling bit 7304 at a desired toolinglocation of the patent, a system can track the location of depth aidelement 7312 to monitor drilling depth of tooling bit 7304.

Depth cameras 7314 are configured to capture one or more images of thedepth aid element 7312. For example, depth cameras 7314 may use multipleimages from multiple cameras that are positioned at fixed and knownlocations relative to one another, and parallax calculations can beperformed to determine the depth of depth aid element 7312. Prior tosuch depth tracking, a registration process may be performed so thatdepth cameras 7314 are properly registered to depth aid element 7312.Additional details on the depth aid registration process are describedin greater detail below. In any case, the depth of depth aid element7312 can be determined and used as a proxy for determining depth oftooling bit 7304. One or more processors (not shown in FIG. 73) can beconfigured to determine the depth of tooling bit 7304 along therotatable axis, based on analysis of the images captured by depthcameras 7314, e.g., by performing the parallax calculations describedabove or other depth determination processing.

As one example, in order to position depth aid element 7312 at a known,fixed distance from tooling bit 7304, rotatable shaft 7302 may beconfigured to include a first portion 7306 that has a different diameterthan a second portion 7310. The different diameters of first portion7306 and second portion 7310 can create a mechanical stop for securingthe position of depth aid element 7312. For example, depth aid element7312 may include a hole that is sized similar to (and slightly largerthan) a diameter of first portion 7306 and smaller than second portion7310. In this way, depth aid element 7312 can be positioned properlysuch that it remains a known, fixed distance from tooling bit 7304 alongrotatable shaft 7302. In other examples, however, a wide variety ofother structures or techniques may be used to ensure that depth aidelement 7312 is secured a known, fixed distance from tooling bit 7304along rotatable shaft 7302. In some examples, a ring of ball bearingsmay be positioned at the physical point of interaction between depth aidelement 7312 and second portion 7310 so as to reduce or substantiallyeliminate friction between depth aid element 7312 and second portion7310. In some cases, depth aid element 7312 may be rotationally fixed,and in other cases, depth aid element 7312 may rotate along withrotating shaft 7302.

In order to facilitate good depth tracking of depth aid element 7312 bydepth cameras 7314, depth aid element 7312 may be designed to includeone or more spherically shaped elements (i.e., one or more sphericalelements) or/or one or more cylindrically shaped elements (i.e., one ormore cylindrical elements). Some or all of depth aid element 7312 maycomprise a spherical or cylindrical element, or in some cases, multiplespheres and/or cylindrical elements may be included on depth aid element7312. Substantially oversized spherical elements and/or cylindricalelements may facilitate better depth tracking than other shapes,although other shapes and sizes may be used. The spherical and/orcylindrical elements may be partially spherical, partially cylindrical,fully cylindrical, fully specially. Other examples of shapes that may bedesirable for depth aid element may include cone shapes. The conical,spherical and/or cylindrical shapes may extend in a direction parallelto rotatable shaft 7302.

Also, it may be desirable for depth aid element 7312 to be relativelylarge in order to ensure that it can provide good depth trackingcapabilities. For example, it may be advantageous to ensure that depthaid element 7312 is an oversized element to help with depth tracking,and e.g., it should usually be larger than other components of medicaldevice 7300, such as the diameter of rotatable shaft 7302 and thediameter of tooling bit 7304. Additional details and desirable shapes ofdepth aid element 7312 are described in greater detail below.

In other examples, in order to facilitate good depth tracking of depthaid element 7312 by depth cameras 7314, depth aid element 7312 may bedesigned to include a planar surface that defines a plane that isperpendicular to an axis that is parallel to rotatable shaft 7302. Insome examples, a planar surface may define a good feature for depthtracking, and in some examples, a planar surface may work better thanspheres or cylinders. In some examples, depth aid element 7312 mayinclude one or more planar surfaces in one or more planes that areperpendicular to an axis that is parallel to rotatable shaft 7302. Insome examples, depth aid element 7312 may include one or more planarsurfaces in combination with one or more cylindrical shapes and/or oneor more spherical shapes. As a non-limiting example, the depth plane ordepth aid elements may be sized in a range of approximately 15-30 mm.

FIG. 74 is a conceptual side view of a portion of a medical device 7400consistent with an example of this disclosure. Medical device 7400comprises a rotatable shaft 7402 and a tooling bit 7410 located on adistal end of rotatable shaft 7402. In this example, tooling bit 7410may comprise a reaming element, e.g., designed for reaming a glenoidbone of a patient to remove bone and prepare the glenoid surface for ananatomical shoulder reconstruction surgery. The opposing proximal end7408 of rotating shaft 7402 may be coupled to a medical reaming toolthat rotates shaft 7402 during the procedure.

As shown in FIG. 74, medical device 7400 includes a depth aid element7412 positioned at a fixed location relative to an axis of rotatableshaft 7402. Like in FIG. 73, in FIG. 74, since depth aid element 7412 ispositioned at a fixed location relative to tooling bit 7410 along anaxis of rotatable shaft 7402, depth aid element 7412 moves co-linearlywith tooling bit 7410 along the axis. In other words, depth aid element7412 is positioned at a known, fixed distance from tooling bit 7410along rotatable shaft 7402. Therefore, tracking of the location of depthaid element 7412 along the axis can be used as a proxy for tracking ofthe location of tooling bit 7410. Accordingly, upon placing tooling bit7410 at a desired tooling location of the patent, e.g., at a location ofa patient's glenoid bone for purposes of glenoid reaming, a system cantrack the location of depth aid element 7412 to monitor reaming depth oftooling bit 7410. In some cases, a guide pin may be installed on thepatient's glenoid bone to orient and guide the reaming process (e.g., inreaming guide pin insertion process of action 1910 of FIG. 19). Thereaming depth, for example, may correspond to the depth into the bonewhere bone is removed from the glenoid.

Depth cameras are not shown in FIG. 74, but like FIG. 73, such depthcameras may be used to capture one or more images of the depth aidelement 7412 as described above with regard to FIG. 73. One or moreprocessors (not shown in FIG. 73) can be configured to determine thedepth of tooling bit 7410 along the rotatable axis, based on analysis ofthe images captured by depth cameras. The processors, for example, maybe processors of an MR device, such as microprocessor 515 ofvisualization device 213 of MR system 212 described herein.Alternatively, processing could be performed remotely by a local orremote computer or a so-called “cloud computer” connected to the depthcameras by a network.

In the example of FIG. 74, in order to position depth aid element 7412 aknown, fixed distance from tooling bit 7410, rotatable shaft 7402 may beconfigured to include a stopper 7404 having a larger diameter than therest of rotatable shaft 7402. Stopper 7404 creates a mechanical stop forsecuring the position of depth aid element 7412 along rotatable shaft7402. For example, depth aid element 7412 may include a mechanicalconnection portion 7406 that includes a hole that is sized similar (andslightly larger than) a diameter of rotatable shaft 7402. Stopper 7404holds depth aid element 7412 in place with the aid of gravity or withthe additional aid of a locking mechanism (not shown). In this way,depth aid element 7412 can be positioned properly such that it remains aknown, fixed distance from tooling bit 7410 along rotatable shaft 7402.

Rotatable shaft 7402 may be free to rotate within the hole definedthrough depth aid element 7412 such that depth aid element 7412 staysrotationally fixed and does not rotate when rotatable shaft 7402rotates. Or in some cases, depth aid element 7412 may rotate with therotation of shaft 7402. However, rotation of depth aid element 7412 maynot be desirable, so additional mechanical elements or stops (not shown)may also be used to ensure that depth aid element 7412 is not allowed torotate when shaft 7402 rotates. In some examples, a ring of ballbearings or other types of friction-reducing elements may be positionedat the physical point of interaction between depth aid element 7412 andstopper 7404 so as to reduce or substantially eliminate friction betweendepth aid element 7412 and stopper 7404 especially when rotatable shaft7402 is rotating.

FIG. 75 is a conceptual perspective view of a mixed reality (MR) systemand example reaming tool 7500 that includes a depth aid element 7502consistent with an example of this disclosure. Reaming tool 7500 maycomprise a reaming bit 7504 that rotates about an axis defined byrotatable shaft 7506. Shaft 7506 may rotate within sleeve 7508. Depthaid element 7502 may be secured to sleeve 7508 so that it is fixed at aknown distance from a reaming surface of reaming bit 7504. Sleeve 7508may be connected to the housing of a surgical drill, and thus, sleeve7508 may be considered part of such surgical drill housing. By securingdepth aid element 7502 to sleeve 7508, depth aid element can remainrotationally fixed when shaft 7506 rotates. At the same time, depth aidelement 7502 will remain a fixed distance from reaming surface ofreaming bit 7504 during the reaming process.

Depth cameras 7510 are configured to capture one or more images of thedepth aid element 7502. For example, similar to the example describedabove, depth cameras 7510 may use multiple images from multiple camerasthat are positioned at fixed and known locations relative to oneanother, and parallax calculations can be performed to determine thedepth of depth aid element 7502. In this way, the depth of depth aidelement 7502 can be determined and used as a proxy for determining depthof a reaming surface of reaming bit 7504. One or more processors (notshown in FIG. 75) can be configured to perform the parallax calculationsto determine the depth of the reaming surface of reaming bit 7504 alongthe rotatable axis defined by shaft 7506, based on analysis of theimages captured by depth cameras 7510. The processors, for example, maybe processors of an MR device, such as microprocessor 515 ofvisualization device 213 described herein. In this way, depth-basedtracking of reaming bit 7504 can be facilitated when reaming a surface7512 of a glenoid bone 7514 of a patient. In some examples werevisualization device 213 is used for depth tracking, audible, tactile orvisual feedback, for example, may be indicative of depth.

Optionally, the procedure may also make use of a mechanical jig 7516,which may be a guide pin that is secured to the patient's glenoid 7514.Mechanical jig 7516 may be inserted with the aid of a surgical drill,and it may provide a mechanical mechanism for guiding the reamingprocess and possibly controlling reaming depth of reaming bit 7504. Insome examples, depth aid element 7502 may be used in combination with ajig-based approach to glenoid reaming (as shown in FIG. 75). In otherexamples, however, depth aid element 7502 may facilitate the eliminationof jig 7516 in favor of depth control based only on depth detection,rather than mechanical limitation. In other words, the use of depth aidelement 7502 may facilitate the glenoid reaming procedure and depthcontrol without the need for a mechanical jig 7516.

FIG. 76 is a conceptual perspective view of a mixed reality system 1051that makes use of an example reaming tool 7602 that includes a depth aidelement 7604 consistent with an example of this disclosure. Reaming tool7602 may comprise a reaming bit 7606 that rotates about an axis definedby rotatable shaft 7608. Shaft 7608 may rotate within sleeve 7610. Depthtracking element 7604 may be secured to sleeve 7610 so that it is fixedat a known distance from a reaming surface of reaming bit 7606. Sleeve7610 may be connected to the housing of a surgical drill, and thus,sleeve 7610 may be considered part of such surgical drill housing. Bysecuring depth aid element 7604 to sleeve 7610, depth aid element canremain rotationally fixed when shaft 7608 rotates. At the same time,depth aid element 7604 will remain a fixed distance from reaming surfaceof reaming bit 7606 during the reaming process.

The example shown in FIG. 76 may utilize a visualization device 213,which may comprise a mixed reality (MR) device such as a mixed realityheadset or goggles worn by a surgeon or other user, that includeadditional depth cameras. In some examples, the depth cameras may beincluded as an internal part of visualization device 213 and in otherexamples, depth cameras may be external to the visualization device soas to provide better depth tracking capabilities.

According to this disclosure, depth cameras 532 (FIG. 5) ofvisualization device 213 or other depth sensors are configured tocapture one or more images of the depth aid element 7604. For example,similar to the other examples described herein, depth cameras 532 mayuse multiple images from multiple cameras that are positioned at fixedand known locations relative to one another, and parallax calculationscan be performed to determine the depth of depth aid element 7604. Inthis way, the depth of depth aid element 7604 can be determined and usedas a proxy for determining depth of a reaming surface of reaming bit7606. Microprocessor 515 (FIG. 5) of visualization device 213 may beconfigured to determine the depth of the reaming surface of reaming bit7606 along the rotatable axis defined by shaft 7608, based on analysisof the images captured by depth cameras 532. The images captured bydepth cameras 532 may contain depth information. In particular,microprocessor 515 may perform parallax calculations based on analysisof the images captured by depth cameras 532 in order to determine depthof depth aid element 7604. In this way, depth-based tracking of reamingbit 7606 (by virtue of tracking depth aid element 7604) can befacilitated in a mixed reality environment when performing a reamingprocedure on a surface 7612 of a patient's glenoid 7614. In someexamples were visualization device 213 is used for depth tracking,audible, tactile or visual feedback, for example, may be indicative ofdepth. For example, visualization device 213 may provide or issue alertsto a user, and such alerts may be indicative of depth. The alerts may begraphical, color-based, color changing, symbols, shape or size-changingsymbols, textual, visual, audible, tactile, or other types of alerts.

In still other examples, a medical device system may be configured toautomatically control the medical device to limit rotation of arotatable shaft based on the depth of the tooling bit along the axisrelative to a desired depth of the tooling bit for the medicalprocedure. For example, drilling, reaming or tooling by the medicaldevice may be automatically disabled once the system detects that depthof the tooling bit has reached a desired depth. In some examples, themedical device system may be configured to control the medical device orother medical devices using the closed loop control techniques describedwith respect to FIG. 72 and elsewhere in this disclosure.

As noted, the procedure may also make use of a mechanical jig 7616secured to the patient's glenoid 7614. Mechanical jig 7616 may comprisea guide pin, and it may provide a mechanical mechanism for guiding thereaming bit 7616, and optionally for controlling or limiting reamingdepth of reaming bit 7616. In some examples, depth aid element 7604 maybe used in combination with a jig-based approach to glenoid reaming (asshown in FIG. 76). In other examples, however, depth aid element 7604may facilitate the elimination of jig 7616 altogether, in mixed realitysystem 7600. In other words, the use of depth aid element 7604 and amixed reality visualization device 213 with depth cameras 532 mayfacilitate the glenoid reaming procedure and depth control without theneed for a mechanical jig 7616.

FIG. 77 is a flow diagram illustrating an example process consistentwith an example of this disclosure. FIG. 77 will be described from theperspective of the system shown in FIG. 73, although the same principlesmay apply to other systems described herein. As shown in FIG. 77, asurgeon may position tooling bit 7304 at a tooling location within apatient (7700). As noted, tooling bit 7304 may comprise a drillingelement, a reaming element, or any of a wide variety of tooling elementsdesigned to rotate about an axis. Indeed, the technique of FIG. 77 maybe particularly well suited for a reaming procedure performed on apatient's glenoid bone with the aid of a mixed reality system.

In some cases, a mechanical jig (e.g., a guide pin) may be inserted intoa patient's glenoid to define positioning of the tooling bit 7304. Inany case, once tooling bit 7304 is placed at the desired location(7700), e.g., a starting location, the depth of depth aid element 7312along an axis of rotatable shaft 7302 can be registered by depth cameras7314 (7702). The registration process may simply be a process ofidentifying a starting location of depth aid element 7312, or in someexamples, the registration process may include a process of registeringa virtual element to depth aid element 7312. After defining the startinglocation of tooling bit 7304 at a tooling location, that startinglocation can also be set or defined for depth aid element 7312 via aregistration process performed by a visualization device 213 such asthat described herein. Then, after the registration process,visualization device 213 can track the depth of depth aid element 7312as a proxy for defining the depth of tooling bit 7304.

Registration process (1104) may be a process of identifying three ormore points so as to define a depth plane associated with depth aidelement 7312. For example, a user using visualization device 213 mayselect three or more points in space to define a plane, and theninitiate a point matching algorithm by visualization device 213 todefine the plane as a virtual plane. In some examples, the user mayselect the points in various ways, such as one or more of hand gestures,a virtual interface, gaze selection, voice commands, or other inputtechniques.

After visualization device 213 performs the point matching algorithm todefine the plane, visualization device 213 may be able to track thatplane by tracking the three or more identified points with depthcameras.

FIGS. 80, 81, and 82 are additional illustrations showing one exampleregistration process registering a location of a depth aid element. Inparticular, FIG. 80 is an example illustration of “best line fitting”whereby observed points 8000 along a line can define the line accordingto a best fit. Upon identifying observed points 8000, which may berepresented by d₀, d₁, d₂ . . . d_(n), visualization device 213 maysearch a line that will minimize the function √{square root over (d₀²+d₁ ²+d₂ ²+ . . . d_(n) ²)}. To define a depth plane, in an analogousmanner to the best line fitting, upon identifying three or more observedpoints to define a plane, visualization device 213 may search a planethat will minimize a planar function.

FIG. 81 shows a scene (schematically) of a humerus 8100 with aninstalled reamer 8102 and depth aid element as a plane 8104. Exemplarydepth camera(s) 8106 are also shown, which in some examples, may be partof or integrated with visualization device 213. Depth camera(s) 8106measure a distance to every pixel in its field of view, and in this way,depth camera(s) 8106 can obtain a point cloud that represents thegeometry of an observed scene.

Depth camera(s) 8106 can generate, as output to visualization device213, a point cloud (and no classification may be done at this stage).Once visualization device 213 has this point cloud, visualization device213 can apply an algorithm that will classify the points in the pointcloud. As an illustrative example, FIG. 82 illustrates various pointclouds that can be observed by depth camera(s) 8106 and defined byvisualization device 213. The illustrated point clouds shown in FIG. 82include a point cloud observed with a depth camera, which may be pointsthat belong to an observed humeral head). The illustrated point cloudsshown in FIG. 82 may also include a point cloud that belongs to a reameraxis, a point cloud that belongs to a reamer back side, and a pointcloud that belongs to the depth aid element (plane).

Visualization device 213 may be configured to refine the classificationsand determine only the points that belong to the depth aid element(plane). Once visualization device 213 determines these points, thatbelong to the depth aid element (plane), visualization device 213 canfit a plane to these points (e.g., according to a best fit planefunction or equation). Again, FIG. 80 is an example illustration of“best line fitting” whereby observed points 8000 along a line can definethe line according to a best fit. A best fit plane can be defined in asimilar way. Once visualization device 213 determines the best fit planefor the first time, visualization device 213 can perform a simpletracking algorithm that will determine the best fit plane at each framethat comes from the depth camera(s).

In another possible example of depth aid registration, the registrationprocess for registering a starting location of depth aid element 7312may be similar registration process for registering a virtual image of apatient's glenoid bone to the actual glenoid bone of the patient, asdescribed elsewhere in this disclosure. According to FIG. 77, in thisexample, depth cameras 7314 of FIG. 73 may be part of a mixed realitysystem that performs the registration process of depth aid element 7312.In some examples, depth tracking of depth aid element 7312 occursrelative to a starting location identified by registering depth aidelement 7312, while in other cases, depth tracking may be performed tomonitor actual depth of depth aid element 7312 relative to the depthcameras, which may not always require registration of depth aid element7312. Moreover, although described primarily in the context of shouldersurgery and reaming of a patient's glenoid bone, the depth trackingtechniques of this disclosure may also be used for other surgicalprocedures, such as for one or more steps of an ankle surgery.

In some examples, the registration process may set and adjust a virtualelement via “SET” and “ADJUST” techniques similar to those describedelsewhere in this disclosure. Using a visualization device 213 a usermay initiate a “SET” to present a virtual element, which may be avirtual version of the depth aid element that defines a depth plane, avirtual version of a backside of a reaming bit that defines a depthplane, a virtual version of a medical device housing or surgical drillthat defines a depth plane, a virtual version of another portion of amedical drill that defines a depth plane, or possibly just a depth planethat is perpendicular to an axis of the tooling bit. The user mayimplement an “ADJUST” technique to adjust the position of the virtualelement, e.g., using gesture-based controls in the mixed reality system.Once placed and adjusted to a desired location, the process may thenimplement a matching algorithm to “MATCH” the virtual element to acorresponding real-world element. In this way, a virtual element usefulfor depth tracking can be registered to a real-world element, such as adepth aid element 7312.

Regardless of how the registration is performed, once the depth of depthaid element 7312 is registered (7702) with the tooling bit positioned ina location for tooling, the surgeon may then activate the medical deviceattached at proximal end 7308 of rotatable shaft 7302 so as to causerotation of shaft 7302 and thereby perform tooling by tooling bit 7304(7704). The surgeon may exert pressure down the axis of rotatable shaft7302 to cause drilling or reaming by tooling bit 7304. Again, the systemmay be especially useful for reaming of a patient's glenoid bone duringan anatomical shoulder reconstruction surgery, in which case the toolingbit 7304 would be a reaming element rather than a drill bit. Althoughsome aspects of this disclosure have focused on reaming depth, it mayalso be helpful to monitor depth when drilling the hole to receive aguide pin, e.g., depth tracking of the drilling depth for placing thereaming pin. A guide pin could then be placed in the hole and thereaming tool may be used with the guide pin. Other instances of drillingdepth tracking may be also be desirable, e.g., for drilling holes toreceive mounting pegs of the glenoid plate, or for drilling, reaming orother tooling of the talus and/or tibia in ankle surgery.

In any case, while performing the tooling by tooling process (7704),depth cameras 7314 can be used to track the position of depth aidelement 7312 relative to an axis of rotatable shaft 7302 (7706). In thisway, the reaming depth (or drilling depth) of tooling bit 7304 can betracked and monitored by tracking the position of depth aid element7312.

In some examples, the process may also identify a desired depth oftooling bit 7304 based on depth tracking of depth aid element 7312(7708). For example, referring to FIG. 76, depth tracking by depthcameras 532 of visualization device 213 may be performed with the aid ofuser interface 522, which may present mixed reality information to auser indicative of depth of a tooling bit (such as reaming element 7606)during the medical procedure. In some examples, user interface 522 maypresent an indication of current actual depth of reaming element 7606.In some examples, user interface 522 may present an indication of adesired depth of reaming element 7606, which may be determined by apreoperative plan. The desired depth may be shown relative to the actualcurrent depth, and in some examples, user interface 522 may present anindication of the remaining depth needed to achieve the desired depth.Audible, tactile or visual feedback, for example, may be indicative ofdepth.

Other mechanisms for medical device tool control (such as describedelsewhere in this disclosure) may also be used to help a surgeon achievea desired depth and help avoid over drilling or over reaming. Forexample, audible or visual cues may be presented to the surgeon viavisualization device 213 based on the depth tracking of depth aidelement 7604. Smart control of the medical device may also beimplemented based on the depth tracking of depth aid element 7604. Insome example, the medical reaming device may be disabled if reamingdepth is determined to meet or exceed the desired depth defined by thepreoperative plan. These or other cues or controls may be used based onthe tracking of depth aid element 7604 in order to improve the medicalprocedure.

FIG. 78 is a side view of anther depth tracking example, which mayeliminate the need for a specially-designed depth aid element. In thisexample, a visualization device such as visualization device 213 (notshown in FIG. 78) may monitor depth of a back surface (i.e., anon-reaming surface) of a reaming element. The system may be designed totrack (and possibly control) the position of the back side of thereaming element. In this example, a depth plane may be registered to theback side of the reaming element, e.g., via a registration processsimilar to that described herein for registering a depth aid element orfor registering a 3D model to a patient's glenoid bone. In this example,the registration process may involve registering a virtual element tothe reaming element (e.g., by registering a virtual plane or a virtualversion of the reaming element to the backside of the actual reamingelement)

By registering a depth plane to a back side of the reaming element, asshown in FIG. 78, depth tracking of the reaming element may then beachieved. The use of a depth aid element as described herein, however,may be desirable relative to tracking the back side of the reamingelement because blood or other substances may inhibit the ability toaccurately track the back side of the reaming element when reaming isbeing performed on a patient.

Referring again to FIG. 78, in still other examples, depth tracking maybe performed with respect to a surgical motor (or the medical devicethat includes the surgical motor), and the depth of the surgical motor(or medical device) could be used as a proxy for depth of the reamingelement. Like the depth aid elements described herein the surgical motorillustrated in FIG. 78 may be located a fixed and known distance from adistal end of the reaming element. Thus, upon registering and trackingdepth of the surgical motor (which may involve registering a virtualelement of the surgical motor to the real surgical motor), tracking canbe achieved on the reaming element in a similar manner to the depthtracking described herein with respect to the depth aid element. In thisexample, a depth plane may be registered to the surgical motor, via aregistration process as described above. By registering a depth plane toa surgical motor, depth tracking of the reaming element may then beachieved. The example shown in FIG. 76, where depth aid element 7604 issecured to a sleeve 7610 may be considered an example where depth aidelement 7604 is attached to the housing of the surgical tool. Sleeve7610, for example, may form part of the housing of a surgical drill, insome examples.

FIG. 79 is another side view showing an example of a depth aid element7902 attached at a fixed location relative to a reamer 7900. The exampleof FIG. 79 may be consistent with other examples that use an attacheddepth aid element. According to this example, a visualization devicesuch as visualization device 213 (not shown in FIG. 79) may be used toadd a plane marker (or other type of virtual marker element) to a reamer7900 in order to track and control depth of the distal end of thereamer. The virtual element may comprise a virtual plane that isregistered to the depth aid element 7902. Using visualization device213, the system can detect points on the plane with depth cameras andfit the plane to a physical depth aid element 7902 that is attached tothe reamer 7900. In this example, a depth plane may be registered todepth aid element 7902, via a registration process similar to thatdescribed above and also described elsewhere in this disclosure, e.g.,for registering a 3D model to a patient's glenoid bone. By registering adepth plane to depth aid element 7902, depth tracking of reamer 7900 canbe achieved.

In some examples, the disclosure is directed to a number ofautomation-based assistance features and techniques related to surgical,intra-operative workflow guidance, some of which may be designed to aida nurse (or other medical assistant, technician, or surgeon) in anorthopedic surgical procedure, such as an orthopedic joint repairsurgical procedure, such as a shoulder repair surgery. It may bedifficult for an operating room nurse to know all of the details of asurgical procedure, such as the next surgical step or which nextinstrument is needed for the next surgical step, especially if the nursedoes not help with the surgical procedure on a regular basis. Forexample, the nurse may assist with a range of surgical procedures havingdifferent requirements and details for surgical steps and instruments,implants or other surgical items used in those steps. The techniques anddevices of this disclosure may help to automate the process to aid thenurse and help reduce errors in the surgery.

In some examples, the techniques leverage surgical items that aredesigned with advanced features, such as sensors, accelerometers, and/orlight sources. One or more processing devices, such as a processingdevice associated with MR system 212 or another computing system, maytrack the surgical items during the surgical procedure and monitor theiruse. Moreover, the one or more processing devices may help control theadvanced features of the surgical items in assisting the nurse withsurgical item selection as the procedure is performed. For example, aset of surgical items may be illuminated by light sources (e.g., lightemitting diodes) that reside within such surgical items, in order toinform the nurse that such tools are needed for the surgical procedure.Moreover, the lighting may be synchronized with the procedure, such thatsurgical items are illuminated when such surgical items are needed,e.g., at particular points during the course of the orthopedic surgicalprocedure. In some examples, a light source of a selected surgical itemmay be illuminated based on a specified use of the surgical item definedin a surgical plan associated with the surgical procedure.

In other cases, surgical items are identified to the nurse byillumination or by virtual elements presented via mixed reality onvisualization devices such as visualization device 213 of MR system 212.Examples of virtual elements presented via MR to identify surgical itemsmay include highlighting of the identified surgical items using regionsof semitransparent color, virtual circles surrounding identifiedsurgical items, virtual arrows pointing to identified surgical items,virtual outlines around identified surgical items, virtual objects thatblock the view of other non-identified surgical items, increasingopacity over other non-identified surgical items, and so on. In someexamples, one or more surgical items of the surgical procedure may beconnected to a controller (e.g., a device that controls lights on thesurgical items or a mixed reality system). Such connectivity may not berequired in examples where MR is used to identify surgical items. Aftera surgical item is used, the next instrument that is needed for theprocedure may be highlighted, lit, identified by one or more virtualelements presented in MR, or otherwise identified to the nurse in anautomated way. The automated identification of surgical items may besynchronized with the steps in the workflow of the surgical procedure,such that surgical items are identified when such surgical items areneeded, e.g., at particular points during the course of the orthopedicsurgical procedure. In this way, techniques and devices of thisdisclosure may help to automate the process to aid the nurse and helpreduce errors in the surgery.

In some cases, a set of surgical items may be illuminated (e.g., usingMR visualizations, using lights on surgical items, using lights in atray, or using another method or combination of methods) during asurgical procedure to aid the nurse with surgical item selection, andspecific types of illumination may be used for different surgical itemsin the set. For example, previously-used surgical items, currently usedsurgical items, and subsequently needed surgical items of the proceduremay be illuminated with different colors or coloring effects, ordifferent lights or spatial or temporal lighting patterns. Byintegrating light sources directly into surgical items, the lightingaccuracy can be better ensured relative to other techniques that usebacklighting or lighting in trays or tables, in which case the lightingmay be incorrect if the surgical item is moved. Moreover, item trackingcan also help to ensure that item identification is accurate and welldocumented during the surgical procedure. Automated documentation ofsurgical items during their use in a surgical procedure can eliminatethe need for surgeons or nurses to physically track and focus on suchdocumentation, allowing the surgeons or nurses to focus on other aspectsof the procedure.

Indeed, item tracking may also be used to help document and log thesurgical procedure, as well as provide a safety check to ensure that thecorrect surgical item is being used, e.g., relative to a preoperativeplan, which may be a patient-specific surgical plan having one or morefeatures that are specifically prescribed for a particular patient.Object-based mixed reality (MR) detection of surgical items may helpwith surgical item verification. In some examples, surgical item use mayalso be monitored or verified with item tracking that monitorsaccelerometer data of accelerometers in the surgical items. In somecases, item tracking may be used in combination with other item trackingor other item registration techniques, such as bar code or other opticalcode scanning, RFID reading or other automation, e.g., using opticalcode scanners, RFID readers or other machine automation tools. In somecases, optical code reading, RFID reading, or other machine automationmay be incorporated into a visualization device, like visualizationdevice 213 in order to allow such automation in an MR environmentwithout the need for additional bar code readers or RFID readers. Inthis way, surgical procedure documentation may be improved by trackingand verifying that the correct surgical items are used at the properstates of the procedure. In order to facilitate optical code scanning ofsurgical items by visualization device 213, some or all of the surgicalitems may include optically scannable codes, such as one-dimensional barcodes or two-dimensional bar codes, which may be printed on the surgicalitems or affixed to the surgical items. Visualization device 213 maydetect a particular surgical item and record its use in a medicalprocedure based on detecting the optically scannable code. Likewise,boxes of surgical items or trays may include optically scannable barcodes that may be scanned by visualization device 213. In some examples,instead of or in addition to the use of optical codes, visualizationdevice 213 may use computer image recognition and/or optical characterrecognition to identify surgical items.

In some examples, some or all of the techniques of this disclosure maybe implemented with the aid of mixed reality (MR) visualization. Forexample, rather than using lights that are integrated into surgicalitems themselves, virtual information may be presented by avisualization device at a position on or adjacent to the real-worldsurgical items viewable by a user via the visualization device, such asa mixed reality headset worn by the nurse during the operatingprocedure. An example of such a MR headset is visualization device 213of MR system 212, e.g., as shown in FIGS. 2 and 5. Moreover, the use ofMR may enable the identification of surgical items into which it may beimpractical to integrate lights, such as implantable devices, woundclosure products, and other types of surgical items. This may eliminatethe need for lighting within surgical items themselves, althoughlighting within certain surgical items may still be desirable even whenMR is used. In other words, in some examples, the MR implementations maybe used in place of physical lighting in surgical items, although inother examples, mixed reality implementations may be used in combinationwith physical lighting in the surgical items. Moreover, mixed realitymay be used in combination with other advanced features in the surgicalitems, such as lights, sensors, accelerometers or other trackingfeatures.

In some examples, a user may view a real-world scene including theenvironment of an operating room, the patient's actual anatomy, i.e.,real anatomy, such as actual bone or soft tissue anatomy (e.g., glenoidor humerus bone in a shoulder arthroplasty) that is exposed duringsurgery. The user may view this scene through a transparent screen,e.g., through see-through, transparent holographic lenses, of ahead-mounted MR device, such as visualization device 213 described inthis disclosure, and the user may also see a virtual MR object orobjects projected on or in the screen, such that the MR object(s) appearto be part of the real-world scene, e.g., with the virtual objectsappearing to the user to be in overlay or otherwise integrated withinthe actual, real-world scene. For example, as discussed above, virtualinformation may be presented or overlaid or presented adjacent to aparticular real-world surgical item, in order to identify the status ofthe surgical item in the surgical procedure, e.g., as a surgical item tobe used in a current step of surgical procedure, a previously-usedsurgical item, or a surgical item to be used in the next step or afuture step of the surgical procedure. The virtual information may beprojected as holographic imagery on the screen of an MR visualizationdevice, such as visualization device 213, e.g., via holographic lenses,that a holographic image is overlaid on a real-world surgical itemvisible through the screen.

In addition to aiding a nurse with surgical item selection, thetechniques and features described herein may also improve surgical itemtracking and documentation of the surgical procedure. As previouslydiscussed, surgical items may include tools, implantable devices, traysof other surgical items, or other physical objects that may be usedduring a surgery. Moreover, the system may be communicatively coupled toan inventory of surgical items that are available. In this case, if asurgeon needs a surgical item that is not present or if the surgeonneeds to change one or more steps of the procedure relative to apreoperative plan, features described herein may allow the nurse tocheck hospital inventory to see if other surgical items are immediatelyavailable for use in the procedure. If so, the nurse may direct anotherassistant to fetch the necessary surgical item from inventory, whileavoiding the need to send the assistant for a manual check on surgicalitem availability. This can save important time during a procedure andhelp to avoid unnecessary down time.

In some examples, the techniques of this disclosure may facilitate moreaccurate documentation of a surgical procedure and more accuratedocumentation of the surgical items used during that procedure. With theuse of a visualization device and mixed reality, a nurse may be able toautomatically document surgical item use. Object identification andobject tracking of surgical items may be performed by a computing systemof the visualization device to help track surgical item selection andsurgical item use. For example, when a surgeon takes a first surgicalitem for use, the system may detect that the surgical item is in thehand of the surgeon, e.g., based on sensor data from the surgical item,image-based object detection, or scanned indicia on the surgical item.Such object tracking (e.g., visual shape detection of surgical item orvisual detection of indicia on the surgical item, or by using sensordata from the surgical item) may be used in place of or in combinationwith other automation, such as optical code scanning, RFID reading, orother item tracking automation. In some examples, object-detection baseditem tracking may be used as a verification and further documentation ofitem use, e.g., in addition to optical code scanning, RFID reading, orother item tracking automation. Again, in some cases, optical codescanning, RFID reading, or other item tracking automation features maybe integrated into an MR device, such as visualization device 213.Visualization device 213 may document surgical item use and save thisdocumentation for record keeping and postoperative analysis. Referringagain to FIG. 6, for example, visualization device 213 may recordtracking information associated with surgical item use into storagedevice(s) 612 for documentation and later retrieval. For example, upondetecting a surgical item or any specific use of a surgical item byvisualization device 213, a time stamp of such detection and the type ofdetection (RFID scan, bar code scan, object detection of the surgicalitem or other automated surgical item detection), the detected surgicalitem and the time stamp associated with that detection can be stored instorage device(s) 612 for documentation of the surgical procedure andlater retrieval. Also, in some examples, one or more surgical items maycommunicate with visualization device 213, e.g., via communicationdevice(s) 616 and a corresponding communication device or module in thesurgical items, so as to convey sensor information, accelerometer data,lighting (as well as time stamps associated with any such information ordata) to visualization device 213 for documentation of the surgical itemuse during the surgical procedure and for later retrieval. In this way,accurate documentation of the surgical procedure may be achieved and theprocedure itself may be improved. Advanced features in the surgicalitem, such as sensors and accelerometers may be used to aid itemtracking so as to verify and document when surgical items are used inthe procedure by monitoring accelerometer data.

FIG. 83 is a block diagram illustrating a system 8300 comprising a setof surgical items 8302 and a processing device 8304 in accordance withan example of this disclosure. Surgical items 8302 may be tracked usingitem tracking techniques of this disclosure. In this example, each ofsurgical items 8302 includes one or more light sources shown as “L(s).”The light sources L(s) may comprise light emitting diodes (LEDs),although other types of semiconductor light elements or other types oflight sources may also be used for light sources L(s). Semiconductorlighting such as LEDs may be desirable because of factors such as lowenergy use, high efficiency, long life, and high intensity. Moreover,lighting effects such as different colors, intensities or pulsing orflickering effects may be easy to control and achieve with LEDs, whichmay comprise multi-color LEDs capable of illuminating lights ofdifferent colors. In some examples, colors may be selected for use inidentifying surgical items for a given environment. For the surgeryroom, for example, it may be desirable to avoid the color red and thecolor green may be very visible for item identification. The LEDs mayhave associated LED controllers configured to drive the LEDs.

Light sources L(s) may be mounted on or within each of the surgicalitems within the set of surgical items 8302. Each surgical item mayinclude a battery, an antenna and communication circuity, e.g., withineach of the surgical item in order to power the light, and provide forcommunication capabilities with processing device 8304, which maycontrol the lights. The lights may be disposed within a cavity orcompartment of a housing of a surgical item. The lights may be part of alight module that includes communication circuity, an antenna andbattery. The lights may be positioned or located for effective lightingof the respective surgical items during the surgical procedure, e.g., ona top side, bottom side, or any desirable location. In some examples, alighting module may be disposably or permanently affixed on the surgicalitem. The lights or lighting modules may be designed to withstandsterilization. Acoustic or vibratory feedback modules (e.g., to “buzz”the surgical item with tactile feedback or to “beep” the surgical itemswith audio feedback) may also be used in addition to or instead oflights. If desired, a surgical item may also include an on/off switch toenable the selectable lighting when the surgical item is in use, and todisable the lighting, when the surgical item is not in use.

FIG. 104 is a more detailed block diagram of a surgical item 10400,which may correspond to one or more of the surgical items described inthis disclosure. Surgical item 10400 may comprise one or more processors10402 and one or more storage devices 10422 communicative coupled via acommunication bus 10424. Surgical item 10400 may further include one ormore sensors 10404 such as any of those described herein for surgicalitem use, surgical item automation, or item tracking. In some examples,surgical item 10400 is illustrated as including one or more temperaturesensors 10406, one or more pressure sensor(s) 10408, one or more motionsensor(s) 10410, and one or more orientation sensor(s) 10412. Surgicalitem 10400 may further include one or more light emitting diode(s) 10420and one or more accelerometer(s) 10418. Surgical item 10400 may alsoinclude a power supply 10416, such as a battery, that powers thecomponents illustrated in surgical item 10400. In addition, surgicalitem 10400 may also include a communication device 10414, which maycomprise a wireless communication device (e.g., with an antenna) forcommunicating with an external device, such as communication device(s)616 of visualization device 213. Sensors 10404 may sense various things(e.g., temperature, motion, pressure, orientation, etc.) as indicationsof surgical item use, and sensed data (and associated timinginformation, such as time stamps associated with the sensed data) can bestored in storage device(s) 10422 and possibly communicated tovisualization device 213, e.g., via wireless communication betweencommunication device(s) 10414 of surgical item 10400 and communicationdevice(s) 616 of visualization device 213.

In general, a device may include a surgical item for use in a surgicalprocedure, and a light on or within the surgical item, wherein the lightis controllable by an external device so as to identify the surgicalitem for use in the surgical procedure. In some examples, the device mayfurther include an accelerometer and/or one or more sensors. Asexamples, the surgical item may comprise a sounder, a compactor, apunching tool, a rasping tool, or a surface planing tool. Alternatively,the surgical item comprises a surgical implant, such as a glenoidimplant, a humeral implant, a fixation device, or anchoring hardware.The surgical item may be a patient-specific surgical item that includesone or more features designed for an anatomy of a specific patient.

In some examples, the set of surgical items 8302 may define a kit ofsurgical items that are usable for a surgical procedure. For example, akit may comprise a plurality of surgical items for use in a surgicalprocedure, and a light on or within each of the plurality of surgicalitems, wherein the light is controllable by an external device so as toselectively identify each of the plurality of surgical items for use inthe surgical procedure. In some examples, each of the plurality ofsurgical items may include an accelerometer and/or one or more sensors.As examples, the kit may include a plurality of sounders, a plurality ofcompactors, a plurality of punching tools, a plurality of rasping tools,or a plurality of surface planing tools. Also, in some cases, at leastone of the surgical items may comprise a surgical implant, such as aglenoid implant, a humeral implant, a fixation device, or anchoringhardware. The surgical item may comprise a patient-specific surgicalitem that includes one or more features designed for an anatomy of aspecific patient. Surgical items 8302 may be defined for a givenprocedure, such as a specific type of shoulder surgery or a specifictype of ankle surgery. Accordingly, the items within surgical items 8302may vary depending on the procedure to be performed.

The term “medical tools” may include any of a wide variety of tools usedin an orthopedic surgical procedure, including for example, sounders,compactors, punching tools, rasping tools, surface planing tools, andother tools used in surgical preparation and implantations. In addition,the term “surgical items” may include medical tools as described herein,surgical implants, and other items that may be used in a surgery. Insome examples, surgical items may be patient-specific. In other words,the item tracking techniques described herein (e.g., using MR, usingadvanced features and lighting, or other features) may also apply tosurgical items in the form of surgical implants, including such thingsas glenoid or humeral implants and associated fixation devices,anchoring hardware, or other implants or tools used to implant suchdevices, e.g., in the example of shoulder joint repair surgery.

As shown in FIG. 83, processing device 8304 is external to the set ofsurgical items. Processing device 8304 is configured to communicate withthe set of surgical items and control the light source in each of thesurgical items based on a surgical procedure. For such communication,processing device 8304 and each of the surgical items in the set ofsurgical items 8302 may include wireless communication capabilitiessupported by wireless communication hardware components, e.g., such as atransmitter, receiver and modem. Any of a wide variety of communicationtechniques may be used, although high-speed short-range communicationprotocols may be especially useful, such as Bluetooth, Zigbee, wirelessUSB, or another high-speed short-range communication protocol. Thewireless communication components within the set of surgical items andprocessing device 8304 are not shown in FIG. 83 for simplicity and easeof illustration.

Although not required for all examples, each of the surgical itemswithin the set of surgical items 8302 may also include other advancedfeatures, such as accelerometers (shown as A(s)) and other sensors(shown as S(s)). Accelerometers A(s) and other sensors S(s) within thesurgical items may facilitate and enhance object tracking in someexamples, e.g., by facilitating the ability to identify the location andmotion of such surgical items. Other sensors S(s) may include agyroscope, one or more temperature sensors for tracking ambient roomtemperature or patient temperature during item use, a pressure sensorfor detecting pressure exerted on the surgical item, a fluid sensor fordetecting fluid such as blood, additional accelerometers, or any type ofsensor that is desired for a surgical item. For example, accelerometersA(s) and other sensors S(s) may allow processing device 8304 todetermine when a specific surgical item is being used by the physicianbased on sensor data and accelerometer data. Sensors S(s) may alsoprovide more advanced information about surgical item use, e.g., basedon pressure, temperature or other sensed parameters during a surgicalprocedure. Such sensed data may be stored in the surgical items orpossibly communicated from the surgical items to an external device,such as processing device 8304, visualization device 213 or anotherexternal device.

In some examples, light sources L(s) within each of the surgical itemswithin the set of surgical items 8302 can aid a nurse in identifying asequence of surgical items needed by the physician in a surgicalprocedure. Processing device 8304 may be configured to the activate ordeactivate the light source L(s) in each of the medical surgical item(e.g., Item 1, Item 2, Item 3, . . . Item N) such that differentsurgical items illuminate at different times. Also, in some cases,processing device 8304 may be configured to the activate or deactivatethe light source L(s) in each of the surgical items (e.g., Item 1, Item2, Item 3, . . . Item N) such that different surgical items illuminatewith different colors (and possibly also at different times). A colorcoordination scheme may be defined so that the nurse is able to usecolors to identify which surgical item is being used, which surgicalitems have already been used, and which surgical item is the nextsurgical item needed by the physician in the surgical procedure. Thesequence of surgical items needed over the course of the surgicalprocedure may be set forth in a virtual surgical plan for the patient,which also may be patient-specific and include tools, implants andprocedure sequences that are specific prescribed for the patient.

In some examples, processing device 8304 is configured to control theset the light sources L(s) in each of the surgical items (Item 1-Item N)so as to identify all of the surgical items in the set of surgical itemsand to specifically distinguish one of the surgical items. In otherwords, all of the surgical items (Item 1-Item N) may be illuminated, buta specific one of the surgical items (e.g., item 2) may be distinguishedin some way. In this way, the nurse may be able to identify the entireset of items as well as the specific surgical item that is currentlyneeded. The set of surgical item (Item 1-Item N) may be included withina larger collection of surgical items, such that illumination of the setof surgical items (Item 1-Item N) distinguishes the set of surgicalitems from the larger collection of surgical items, and thedistinguished lighting of the specific surgical item (e.g., Item 2) mayallow the nurse to more quickly identify the specific surgical itemwithin the set.

In some examples, a different light color may be used to distinguish oneof the surgical items from the set of surgical items. In some examples,a lighting effect (e.g., blinking or flashing light, which may definetemporal flashing patters or rates on an item-by-item basis) may be usedto distinguish one of the surgical items. In some examples, a lightingintensity is used to distinguish one of the surgical items. In otherexamples, if multiple lights are provided in a surgical item,illumination of a number of lights or a pattern of lights on thesurgical item may be used to distinguish the surgical item. These orother examples (as well as any combination of such distinguishedlighting) may be used to help the nurse quickly and accurately identifythe specific surgical item within the set

In some examples, the light source L(s) in each of the surgical items iscontrolled to distinguish a first subset of the surgical items, a secondsubset of the surgical items and a third subset of the surgical items.For example, the first subset of the surgical items may correspond toalready used surgical items, the second subset of the surgical items maycorrespond a currently used surgical item, and the third subset of thesurgical items may correspond to subsequently needed surgical items.

In some examples, the set of surgical items 8302 (Item 1-Item N) maydefine similar surgical items of different sizes or shapes, which may beused in succession in the surgical procedure. For example, the set ofsurgical items 8302 (Item 1-Item N) may correspond to a set oforthopedic sounders (e.g., for sounding the humeral canal) of sequentialsizes or shapes for an orthopedic shoulder reconstruction surgery. Insome examples, the set of surgical items 8302 (Item 1-Item N) mayfurther comprise one or more punching tools of sequential sizes orshapes for an orthopedic shoulder reconstruction surgery. In someexamples, the set of surgical items 8302 (Item 1-Item N) may furthercomprise one or more compacting tools of sequential sizes or shapes foran orthopedic shoulder reconstruction surgery. In still another example,the set of surgical items 8302 (Item 1-Item N) may further comprise oneor more surface planing tools of sequential sizes or shapes for anorthopedic shoulder reconstruction surgery. In these and other examples,the set of surgical items 8302 (Item 1-Item N) of different sizes orshapes may be illuminated, but the illumination may differ depending onwhether such surgical items have been previously used, such surgicalitems are in current use, or such surgical items are planned forsubsequent use. The entire set may be illuminated, but the illuminationmay be conspicuously coded so that the nurse or other user is able todistinguish previously-used surgical items from the current surgicalitem in use, and to distinguish subsequently needed surgical items fromthe current surgical item in use and the previously-used surgical items.Different types of light colors, different types of light intensities ordifferent types of lighting effects may be used (possibly incombination) to aid the nurse in surgical item selection. The set ofsurgical items to be used in the procedure may be defined by apreoperative plan, and the techniques of this disclosure may identifyall surgical items of the procedure and may also identify a particularsurgical item that is the current surgical item needed in the procedureaccording to the preoperative plan, e.g., as defined by a virtualpatient-specific surgical plan. In some cases, surgical items may rangein size from e.g., 1 to 10, and the system may be designed to identify asubset of these sizes for use, e.g., according to a preoperative plan.In some cases, there may be one light for an entire set of surgicalitems and another light for a specific surgical item to be used fromthat set.

As mentioned above, in some examples, some or all of the techniques ofthis disclosure may be implemented with the aid of mixed reality. Forexample, rather than using lights that are integrated into the surgicalitems themselves, virtual information may be overlaid on the surgicalitems by a visualization device, such as a MR headset worn by the nurseduring the operating procedure. This may eliminate the need for lightingin surgical items, although lighting may still be desirable even whenmixed reality is used. In other words, in some example, the mixedreality implementations may be used in place of physical lighting in thesurgical items, although in other examples, mixed realityimplementations may be used in combination with physical lighting in thesurgical items. Moreover, mixed reality may be used in combination withother advanced features in the surgical items, such as lights, sensors,accelerometers or other item tracking features.

FIG. 84 is a conceptual block diagram illustrating a medical devicesystem 8400 comprising a set of surgical items 8402 and a processingdevice in the form of an MR visualization device 213 in accordance withan example of this disclosure. In this example, each of the surgicalitems in set 8402 is shown with an overlaid virtual element shown as“V(s).” The virtual elements V(s) may be presented to the user ofvisualization device 213 as virtual elements of a mixed realitypresentation. The virtual elements V(s) may function in a similar mannerto light sources L(s) shown and described in FIG. 83. However, virtualelements V(s) are not really part of the set of surgical items 8402.Instead, they are elements presented by visualization device 213 so asto appear to be associated with each of the surgical items in the set ofsurgical items 8402. In this way, rather than using lights that areintegrated into the surgical items themselves, mixed reality is used topresent virtual information presented or possibly overlaid on thesurgical items by visualization device 213. Visualization device 213 maybe used for surgical item identification, e.g., for a nurse, while alsoproviding intraoperative guidance and workflow guidance. The views seenon visualization device 213 may be different for a nurse relative to theviews of a surgeon. The item identification features or presentation onvisualization device 213 may be presented to a nurse, and not presentedto the surgeon, although view sharing techniques may also be used,whereby a surgeon is able to select and view images or objects from thenurse's view, as described in greater detail elsewhere in thisdisclosure.

According to this disclosure, visualization device 213 presents virtualinformation V(s) on or around each of the surgical items within the setof surgical items 8402 in a way that can aid a nurse in identifying asequence of surgical items needed by the physician in a surgicalprocedure. Such virtual information may include virtual halos, outlines,highlighting, marks, text (including lettering and numbers), or othertypes of virtual elements to distinguish the surgical items.Visualization device 213 may be configured to the activate or deactivatethe virtual elements V(s) in each of the surgical item (e.g., Item 1,Item 2, Item 3, . . . Item N) of the set of surgical items 8402 suchthat different surgical items appear to be illuminated with differentcolors. For instance, differently colored virtual halos, outlining, orhighlighting may be used to illuminate surgical items. A colorcoordination scheme may be defined so that the nurse is able to usecolors to identify which surgical item is being used, which surgicalitems have already been used, and which of the surgical items is thenext surgical item needed by the physician in the surgical procedure.The sequence of surgical items needed over the course of the surgicalprocedure may be set forth in a preoperative plan.

In some examples, visualization device 213 is configured to control thevirtual information V(s) associated with each of the surgical items(Item 1-Item N) within the set of surgical items 8402 so as to identifyall of the surgical items in the set of surgical items and tospecifically distinguish one of the surgical items. In other words, allof the surgical items (Item 1-Item N) within set 8402 may be identifiedby virtual information V(s), but a specific one of the items (e.g., item2 within set 8402) may be distinguished by its associated virtualinformation V(s). The virtual information V(s) may include any of thetypes of virtual information described elsewhere in this disclosure fordistinguishing surgical items. In this way, mixed reality may be used tohelp the nurse to identify the entire set of surgical items as well asthe specific surgical item that is currently needed. The set 8402 ofsurgical items (Item 1-Item N) may be included within a largercollection of surgical items, such that virtual identification of theset of surgical items (Item 1-Item N) with virtual information V(s)distinguishes the set of surgical items from the larger collection ofsurgical items, and the distinguished virtual information of thespecific surgical item (e.g., V(s) of Item 2) may allow the nurse tomore quickly identify the specific surgical item within the set.

In some examples, virtual elements V(s) that are overlaid on eachsurgical item in the set of surgical items 8402 may comprise differentcolors, so as to distinguish one of the surgical items from the set ofsurgical items. In some examples, a MR lighting effect may be used withvirtual elements V(s) (e.g., blinking or flashing light) to distinguishone of the surgical items. In some examples, a lighting intensityapplied to virtual elements V(s) to distinguish one of the surgicalitems. These or other examples (as well as any combination of suchdistinguished lighting by virtual elements V(s)) may be used to help thenurse quickly and accurately identify the specific surgical item withinthe set.

In some examples, a medical device system comprises a plurality ofsurgical items, a visualization device configured to present a MRpresentation to a user; and one or more processors configured to selecta surgical item of the plurality of surgical items based on a surgicalprocedure. The one or more processors may be further configured topresent, in the MR presentation, virtual information that identifies theselected surgical item among the plurality of surgical items. Thevirtual information may be presented on or adjacent a position of theselected surgical item visible via the visualization device. Theprocessors may be part of the MR visualization device or the processorsmay be external to the MR visualization device. As described in greaterdetail elsewhere in this disclosure, the MR visualization device maycomprise one or more see-through holographic lenses and one or moredisplay devices that display an image via the holographic lenses topresent the virtual model and the virtual guide to the user.

As further examples of MR content presented to the user, e.g., byvisualization device 213, virtual elements V(s) may compriseidentification loops or circles circumscribing the surgical item foridentification, arrows, X's, reticles, brackets, shapes (circles,rectangles, squares, ovals, numbers, letters, words or symbols, whichmay flash or have other effects), different background hatching orpatterns, combined with colors or intensities. These and a wide varietyof other types of MR content could be added as visual elements V(s) tohighlight (or distinguish) a given surgical item with the aid of MR.

In some examples, the virtual elements V(s) associated with each of thesurgical items are displayed to the user via visualization device 213 ina way to distinguish a first subset of the surgical items, a secondsubset of the surgical items and a third subset of the surgical items.For example, the first subset of the surgical items may correspond toalready used surgical items, the second subset of the surgical items maycorrespond a currently used surgical item, and the third subset of thesurgical items may correspond to subsequently needed surgical items.

In some examples, the set of surgical items (Item 1-Item N) of set 8402may define similar surgical items of different sizes or shapes, whichmay be used in succession in the surgical procedure. For example, theset of surgical items 8402 (Item 1-Item N) may correspond to a set ofsounders of sequential sizes or shapes for an orthopedic shoulderreconstruction surgery. In addition, the set of surgical items 8402(Item 1-Item N) may further comprise one or more punching tools ofsequential sizes or shapes for an orthopedic shoulder reconstructionsurgery. Moreover, the set of surgical items 8402 (Item 1-Item N) mayalso comprise one or more compacting tools of sequential sizes or shapesfor an orthopedic shoulder reconstruction surgery. In addition, the setof surgical items 8402 (Item 1-Item N) may also comprise one or moresurface planing tools of sequential sizes or shapes for an orthopedicshoulder reconstruction surgery.

In these and other examples, the set of surgical items 8402 (Item 1-ItemN) of different sizes or shapes may be identified with virtual elementsV(s). In some examples, each surgical item is identified when it is timefor that surgical item to be used. However, in some examples, multiplesurgical items may be identified. In some examples, the type of element,color or shape of virtual elements V(s) may differ depending on whethersuch surgical items have been previously used, such surgical items arein current use, or such surgical items are planned for subsequent use.The entire set may be identified by virtual elements V(s), but thevirtual elements V(s) may be conspicuously defined so that the nurse isable to distinguish previously-used surgical items from the currentsurgical item in use, and to distinguish subsequently needed surgicalitems from the current surgical item in use and the previously-usedsurgical items. Different types of colors, different types ofintensities or different types of effects may be used with virtualelements V(s) (possibly in combination) to aid the nurse in surgicalitem selection.

The set of surgical items to be used in the procedure may be defined bya preoperative plan, and the techniques of this disclosure may identifyall surgical items of the procedure and may also identify a particularsurgical item that is the current surgical item needed in the procedureaccording to the preoperative plan. Again, visualization device 213 maybe used for surgical item identification, e.g., for a nurse, while alsoproviding intraoperative guidance and workflow guidance. The views seenon visualization device 213 may be different for a nurse relative to theviews of a surgeon. The item identification features or presentation onvisualization device 213 may be presented to a nurse, and not presentedto the surgeon, although view sharing techniques may also be used,whereby a surgeon is able to select and view images or objects from thenurse's view, as described in greater detail elsewhere in thisdisclosure.

FIG. 85 is another conceptual block diagram illustrating a medicaldevice system 8500 comprising a set of surgical items 8502 and aprocessing device in the form of a visualization device 213 inaccordance with an example of this disclosure. In this example, medicaldevice system 8500 may include some or all of the features describedabove with respect to system 8300 of FIG. 83 in combination with some orall of the features described above with respect to system 8500 of FIG.85. Each of the surgical items in set 8502 may include light sourcesL(s), accelerometers A(s), other sensors (S(s), as well as one or morevirtual elements shown as “V(s).” The virtual elements V(s) may bepresented to the user of visualization device 213 as a virtual elementsof a mixed reality presentation. In this way, surgical items withadvanced features, such as light sources L(s), accelerometers A(s),other sensors (S(s) may be used along with mixed reality and virtualelements V(s) to provide for an advanced set of surgical items withmixed reality controls, item tracking and procedure tracking. The use ofvirtual elements V(s) in combination with light sources L(s) may allowfor item identification by persons wearing or otherwise using avisualization device, as well as persons not using a visualizationdevice. Or light sources L(s) may be used in combination with virtualelements V(s) to improve item identification or different illuminationor identification effects.

In some examples, MR system 212 may be configured to allow for itemtracking with the aid of mixed reality. Moreover, MR system 212 may alsoallow for tracking and documentation of the surgical procedure usingmixed reality. Item tracking, for example, may be performed by MR system212 to help document and log the surgical procedure, as well as providea safety check to ensure that the correct surgical item is being used,e.g., relative to a preoperative plan. MR system 212 may utilizeobject-based detection of the surgical items.

For example, when a physician is given one of the surgical items in theset of surgical items 8502, MR system 212 may perform visual objectdetection as a verification to ensure that the correct surgical item isbeing used. Visual object detection may refer to detection of objectshape (or possibly detection of bar code, text, letters, numbers words,or other identifying indicia associated with a surgical item). Suchvisual object detection may be used in addition to bar code scanning,RFID reading or other item tracking techniques. Moreover, accelerometerdata associated with the accelerometer A(s) of a selected surgical itemmay be used as a further verification and documentation of surgical itemuse. For example, accelerometer data associated with accelerometers(A(s) may be indicative of surgical item use, surgical item resting,specific type of surgical item positioning. Such data may be useful toidentify and document surgical item use (or non-use).

The use of a combination of item tracking techniques may help to improvethe accuracy of item tracking and documentation during the procedure.According to this disclosure, two or more of the following techniquesmay be used to verify surgical item use: accelerometer data, visualobject detection of the surgical item with cameras of an MRvisualization device, tray tracking by an MR visualization device bytracking and monitoring the absence of the surgical item in the trayduring its use so as do identify a likely time of surgical item use,RFID or optical code scanning of the surgical item, or other trackingtechniques. Tray tracking may use cameras to identify the absence of asurgical item from a tray and the absence of the surgical item from thetray may be used to infer that the surgical item is likely in use. Insome examples, three or more, four or more, or even five or more ofthese item tracking techniques may be used to help to ensure that itemtracking is accurate and well-documented during the procedure.

Item tracking information may be recorded by visualization device 213during the procedure as a record of the surgical procedure. The recordeditem tracking information may include video of how surgical items areused, records of which surgical items were used, records of whenindividual surgical items were used, and so on. For instance,information about which surgical items were used can be To facilitatesuch multi-mode tracking of surgical items, in some cases, optical codereading, object detection with the user of cameras, RFID reading orother machine automation may be incorporated into a visualizationdevice, like visualization device 213 in order to allow such automationin an MR environment without the need for additional bar code readers orRFID readers. In this way, surgical procedure documentation may beimproved by tracking and verifying that the correct surgical items areused at the proper states of the procedure.

System 8500 may be configured to track item use and recommend the nextsurgical item to be used for the procedure. For example, system 8500 maybe configured to select a surgical item based on a surgical plan, suchas a preoperatively-defined surgical plan. In this example, the surgicalplan may contain information that specifies which surgical items are tobe used in various steps of the surgery defined by the surgical plan.Accordingly, system 8500 may use such information in the surgical planto select the surgical item based on the surgical plan. Moreover, afterusing all of the surgical items (Item 1-Item N) in the set of surgicalitems 8502, system 8500 may be configured to move on to a next stage ofthe surgical procedure, e.g., possibly identifying an entirely new setof surgical items, e.g., in a different tray or box, for that next step.In some examples, the item tracking and item identification features maybe specific to a nurse wearing or otherwise using an MR device such asvisualization device 213.

The surgeon may also be wearing or otherwise using a similar MR device,but the surgeon's MR device may not include the item tracking and itemidentification features performed by MR device of the nurse. In otherwords, in some examples, the surgeon's MR device be configured toprovide all of the functionality of the nurses' MR device, while inother examples, the surgeon's MR device may not be configured presentvirtual imagery that identifies a particular surgical item and may notinclude visual detection features or features for processing ofaccelerometer data for identifying a surgical item or use of a surgicalitem.

In some examples, view sharing techniques, which are described elsewherein this disclosure, could be used to enable the surgeon to see the viewsof the nurse, but typically, the surgeon may be viewing his or hersurgeon-specific MR presentation and not the item tracking MR elementsseen by the nurse, such that the surgeon may focus attention on theorthopedic surgical procedure and surgical bone or tissue site beingaddressed. The MR elements seen by the nurse for item identification maybe visible to that nurse only when the nurse views the set of surgicalitems.

A record of the surgical procedure can be very helpful or important forsafety tracking, quality control, legal compliance, analysis of thesurgical procedure, tracking of how many times an instrument or othersurgical item has been used, instrument age, instrument longevity, orother reasons. In addition to tracking surgical item use, visualizationdevice 213 (or possibly another device if MR is not used in theprocedure) may be configured to log times associated with surgical itemuse in the procedure, log a counter for each surgical item use, recordtime between steps of the procedure, or other timing or tracking. Forexample, surgical item use and surgical item monitoring may be used torecord the amount of time that the surgeon performed sounding, theamount of time that the surgeon performed rasping, the amount of timethat the surgeon performed compacting, the amount of time that thesurgeon performed surface planing, and so forth. By using surgical itemuse to track each step of the procedure and by recording the time thateach surgical item is used, the system may record a very accuratepicture of the entire surgical procedure.

Other things that may be useful to track may include the number ofsterilizations performed on a surgical item, the number of times that asurgical item is moved, the number of times or the amount of time that asurgical item is used in a surgical procedure, an indication that asurgical item has been dropped on the floor, or other surgicalitem-specific types of use or misuse. In order to record steps of asurgical procedure in a surgical procedure log, advanced surgical itemfeatures, such as light sources L(s), accelerometers A(s), other sensors(S(s), may be used alone or in combination with mixed realitytechniques, including MR-based item tracking using visual elements V(s),visual object detection using sensors mounted on a MR visualizationdevice, and the presentation of virtual information V(s) to the nurse.

For example, surgical procedure recordkeeping and surgical item trackingmay be divided into stages of a surgical procedure. FIGS. 86-94 showsone example of stages of an anatomic arthroplasty whereby item tracking,item selection and identification and documentation of surgical item usemay be performed and recorded for each stage of the procedure. Theexample set forth in FIGS. 86-94 show humeral preparation and implant ina shoulder arthroplasty for purposes of illustration, but similartechniques to those described herein can be applied to other steps in ashoulder arthroplasty such as glenoid preparation and implant, as wellas other orthopedic procedures. Although FIGS. 86-94 as well as FIGS.95-108 discuss aspects of specific shoulder surgery steps, thetechniques of this disclosure may apply to other types of surgery.Moreover, ankle surgery may involves the use of multiple tools and jigsfor drilling, cutting, preparing implant sites and installing implantson a patient's talus and tibia, and the tool tracking techniques of thisdisclosure may be utilized for ankle surgeries, especially when avariety of different tools are used in the procedure.

In the example shown in FIG. 86-94, the procedure may be divided intostages, such as a sounding procedure shown in FIG. 86, a punchingprocedure shown in FIG. 87, a compacting or rasping procedure shown inFIG. 88, a surface planing procedure shown in FIG. 89, a protectprocedure shown in FIG. 90, a trial procedure shown in FIGS. 91 and 92,and an implant procedure shown in FIGS. 93 and 94. Each stage mayinvolve the use of multiple surgical items, and the techniques of thisdisclosure may aid in surgical item selection and surgical itemidentification of the multiple surgical items used in each stage. Insome examples, sets of surgical items are defined for the entireprocedure, while in other examples, sets of surgical items are definedfor each stage of the overall procedure. In either case, the set ofsurgical items may be identified while also distinguishing a particularsurgical item to be used in a current step of the procedure, e.g., asdefined by a preoperative plan. Or the techniques could distinguish asingle surgical item in a set of surgical items without identifyingother surgical items in the set of surgical items, e.g., identifying onesurgical item at a time for each step or stage of the procedure. Thesounding procedure shown in FIG. 86 may involve insertion of one or moresounders 8600 into the soft tissue inside a patient's humeral bone 8602.The punching procedure shown in FIG. 87 may involve connection andinsertion of punching tools 8700 into the patient's humeral bone 8702 inthe area where the sounder was inserted (e.g., shown in FIG. 86). Thecompacting or rasping procedure shown in FIG. 88 may include theinsertion of one or more compacting tools 8800 into the hole created inthe patient's humeral bone 8802. Compacting or rasping may be performedwith a long stem or a short stem, depending on the circumstances. Thesurface planing procedure shown in FIG. 89 may include the surfaceplaning via planing tools 8900 on patient's humeral bone 8902. Theprotect procedure shown in FIG. 90 may include attachment of protectionhardware 9000 to the patient's humeral bone 9002. The trial procedureshown in FIGS. 91 and 92 may involve the attachment of additionalimplantation hardware 9100 or 9200 to the patient's humeral bone 9102 or9202, and the implant procedure shown in FIGS. 93 and 94 may includeattachment of the final implant 9300 or 9400 to the patient's humeralbone 9302 or 9402.

If the procedure is divided into stages, then upon identifying the useof the final surgical item for any given stage, visualization device 213may be programmed with the stages of the procedure and configured toalert the nurse that the given stage is complete and that its time toprepare for the next stage. In this case, the nurse may be prompted toprepare another set of surgical items as the next set of surgical itemsto be used in the surgical procedure. For example, after finishing asounding procedure shown in FIG. 86, which may involve the use of aplurality of differently sized sounders that can be identified andtracked as described herein, the nurse may be prompted by visualizationdevice 213 to prepare the next set of surgical items, i.e., a set ofpunching tools used for the punching procedure shown in FIG. 87.

Similarly, after finishing the punching procedure shown in FIG. 87,which may involve the use of one or more punching tools that can beidentified and tracked as described herein, the nurse may be prompted byvisualization device 213 to prepare the next set of surgical item, i.e.,a set of compacting tools used for the compacting or rasping procedureshown in FIG. 88. Then, after finishing the compacting or raspingprocedure shown in FIG. 88, which may also involve the use of aplurality of different sized compacting tools that can be identified andtracked as described herein, the nurse may be prompted by visualizationdevice 213 to prepare the next set of surgical items, i.e., a set ofsurface planing tools used for the surface planing procedure shown inFIG. 89. Visualization device 213 may aid in surgical item selection,surgical item preparation, and procedure tracking of each step orsub-step of the surgical procedure. Prompting the nurse for each stepmay include the presentation of visual cues, virtual elements or otherMR content on visualization device 213.

FIG. 95 is a conceptual side view of a sounding procedure beingperformed on a human humeral bone in which a sounder 9500 is insertedinto a patient's humeral bone 9502. FIG. 96 is an illustration of a setof medical sounders 9600 of different sizes for use in the surgicalprocedure shown in FIG. 95. Typically, the sounders are used insuccession from a smallest sounder to a largest sounder. Depending onthe preoperative plan, some of the sounders may not be used. Forexample, in some cases the first sounder used may not be the smallestsounder in the set of surgical items, in which case the smallest soundermay not be used in the procedure. Similarly, in some cases, the lastsounder used may not be the largest sounder, in which case the largestsounder may not be used in the procedure. The specific sounders to beused may be defined by a preoperative plan, and the surgical itemidentification techniques may identify surgical items in accordance withthe preoperative plan. In some cases, a preoperative surgical plan mayidentify the surgical items to be used for a given step of a surgicalprocedure, and visualization device 213 may identify the last surgicalitem for a given step of a procedure. Then, upon detection of that lastsurgical item's use, visualization device 213 may automatically identifythe next step of the surgical procedure to the user.

According to this disclosure, advanced features (such as accelerometers,light sources or other sensors) may be included in each of the soundersshown in FIG. 96 to enable item identification and item tracking asdescribed herein. Moreover, in some cases, MR system 212 may presentvirtual elements, e.g., overlaid on or placed on around or adjacent toeach of the sounders shown in FIG. 96, to enable item identification anditem tracking as described herein. Procedure tracking, timing of use,and documentation of surgical item use may also be recorded by MR system212 so as to enable automated surgical item identification, proceduretracking and surgical procedure recording.

FIG. 97 is an illustration of a set of surgical items, which include aset of progressively larger sounders 9700. The techniques of thisdisclosure may take advantage of advanced features (such asaccelerometers, light sources or other sensors) in each of the sounders9700 to enable item identification and item tracking as describedherein. Moreover, in some cases, MR system 212 may present virtualelements on each of the sounders 9700 to enable item identification anditem tracking as described herein.

The set of surgical items illustrated in FIG. 97 may also includepunching tools located next to each size of sounders 9700. According toa surgical procedure, a particular punching tool (e.g., punching tool9702) may be defined by a preoperative surgical plan as corresponding toa largest sounder size used according to the preoperative plan. In otherwords, although the set of surgical items shown in FIG. 97 includesprogressively sized punching tools, only one of the punching tools maybe used in the procedure, e.g., corresponding to the largest sounderused. For this reason, surgical item identification, e.g., by avisualization device 213, to distinguish only that particular punchingtool needed for the punching step can be very desirable to help ensurethat the correct surgical item is used. Moreover, surgical item trackingand verification described herein may help avoid a scenario where anincorrect sized punching tool is used. FIG. 105 is an illustration of aset of surgical items similar to those illustrated in FIG. 97. FIG. 105shows one exemplary virtual element 10500, which may be presented byvisualization device 213, e.g., as an overlay, to identify ordistinguish a particular surgical item needed for a surgical step. FIG.106 is another exemplary illustration of a set of surgical items similarto those illustrated in FIG. 97. FIG. 106 shows another exemplaryvirtual element 10600, which may be presented by visualization device213, e.g., as arrows and/or outlines of a given surgical item, toidentify or distinguish that surgical item needed for the next step ofthe surgical procedure. Although FIGS. 105 and 106 show two examples ofvirtual elements 10500 and 10600, many other types of virtual elementscould be used and presented by vitalization device 213 to identifysurgical items.

FIG. 98 is a conceptual side view of a compacting or rasping procedurebeing performed on a human humeral bone. FIG. 99 is an illustration of aset of surgical items, which include a set of progressively largercompacting tools 9900. The techniques of this disclosure may useadvanced features (such as accelerometers, light sources or othersensors) in each of the compacting tools 9900 to enable itemidentification and item tracking as described herein. Moreover, in somecases, MR system 212 may present virtual elements on each of thecompacting tools 9900 to enable item identification and item tracking asdescribed herein.

Items may also require assembly, e.g., requiring attachment ofcompacting tools 9900 to a handle 9902. Item assembly may be illustratedto the nurse by an instructional diagram, animation or video shown in avisualization device 213 worn by the nurse. The display of theinstructional diagram, animation or video may be initiated by aselecting a widget or icon in an MR or AR display so that the nurse canbe shown how to properly assemble the items, if such instruction isneeded in real time. Instructional diagrams, animations or videos may bepresented to illustrate steps of the procedure, and for a nurse,instructional diagrams, animations or videos to illustrate item assemblycan be very helpful.

Rasping tools 9900 may comprise two sided tools that have both long stemand short stem rasping elements. Accordingly, item identification (e.g.,by lighting or by virtual overlays of virtual elements in mixed reality)may also identify the rasping side that should be used in the procedure(e.g., the long stem or the short stem). This can be very helpful to anurse in order to ensure that long stem rasping is used when long stemrasping is defined in the preoperative plan, or to ensure that shortstem rasping is used with short stem rasping is defined in thepreoperative plan. When each rasping tool is two sided and only one sidemay be used in the procedure, it may be very desirable to identify notonly the rasping tool, but also the portion of the rasping tool to beused, e.g., via lighting within the rasping tool or via mixed realityinformation. Accordingly, the rasping tool may be configured toselectively light one side of the rasping tool to identify that side asbeing the appropriate side for rasping, or visualization device 213 maybe configured to present virtual information that identifies aparticular side of the rasping tool as being the appropriate side forrasping. Surgical item assembly may also be shown (e.g., viainstructional diagram, animation or video in the nurse's mixed realitypresentation) in a way that clearly shows assembly of the rasping toolfor short stem rasping or long stem rasping.

In some examples, e.g., according to a preoperative plan, the itemidentification techniques described herein may identify a particularrasping tool that is three sizes below the final sounder that is used inthe sounding process. Rasping may be sequentially performed untilsatisfactory fixation is achieved. Satisfactory fixation can be assessedby a slight torque motion of the inserter handle. The compactor orrasping tool should not move within the patient's bone. Accordingly,since the final rasping tool size may be undefined by the preoperativesurgical plan and defined or selected during the intraoperative process,a sequence of rasping tools may be identified by the techniques of thisdisclosure with unique coloring, visual effects or even text overlay viaan MR device such as visualization device 213. Moreover, the itemidentification techniques for rasping may explain (with text, animationor videos) or identify to the nurse (via virtual elements) that thefinal rasping tool size is that which achieves a satisfactory level offixation, which may be determined by the physician during the procedure.FIG. 107 is an illustration of a set of surgical items similar to thoseillustrated in FIG. 99. FIG. 107 shows one exemplary set of virtualelements 10700 and 10702, which may be presented by visualization device213, e.g., as overlays, to identify or distinguish a particular surgicalitem needed for a surgical step. Virtual elements 10700 and 10702 mayrequire assembly, and in some cases, visualization device 213 maypresent an instructional video to demonstrate the assembly process(which may be selectable by the nurse and shown only when needed).Furthermore, virtual element 10702 may be configured to not onlyidentify the surgical item, but to identify a side of the surgical item(in this case the long stem of a rasping tool) so as to inform the nurseof the side of a two-sided surgical element that should be used in thesurgical procedure.

FIG. 100 is a conceptual side view of a surface planing procedure beingperformed on a human humeral bone. FIG. 101 is an illustration of a setof surgical items, which include a set of progressively larger surfaceplaning tools 10100. The techniques of this disclosure may takeadvantage of advanced features (such as accelerometers, light sources orother sensors) in each of the surface planing tools 10100 to enablesurgical item identification and surgical item tracking as describedherein. Moreover, in some cases, additionally or alternatively,visualization device 213 may present virtual elements on each of thecompacting tools 9900 to enable item identification and item tracking asdescribed herein.

Surface planing may involve the use of only one of surface planing tools10100, e.g., corresponding to a similarly sized largest sounder used asthe final sounder and corresponding to the punching tool used. For thisreason, surgical item identification to distinguish only that particularsurface planing tool needed for the surface planing step can be verydesirable to help ensure that the correct surgical item is used.Moreover, item tracking and verification described herein may help avoida scenario where an incorrect sized punching tool is used. FIG. 108 isan illustration of a set of surgical items similar to those illustratedin FIG. 101. FIG. 108 shows one exemplary set of virtual elements 10800and 10802, which may be presented by visualization device 213, e.g., asoverlays, to identify or distinguish a set of surgical items needed fora surgical procedure. In this example, virtual element 10802 may beconfigured to distinguish a given surgical item from the otheridentified surgical items 10800 in order to identify that surgical itemas being the one needed in a current surgical step of the surgicalprocedure.

FIG. 102 is a flow diagram illustrating a technique of identifyingsurgical items in a surgical procedure. The method of FIG. 102 may beperformed, for example, with a visualization device 213 that presentsvirtual elements or with the use of surgical items that includeintegrated lighting as described herein. As shown in FIG. 102, a methodincludes visually identifying a set of surgical items (10200), andvisually distinguishing a particular surgical item within the set ofsurgical items (10202). In some examples, visually identifying the setof surgical items may comprise controlling a light source in each of thesurgical items, and visually distinguishing the particular surgical itemmay comprise controlling a particular light source of the particularsurgical item. In this case, visually distinguishing the particularsurgical item may comprise one or more of: controlling the particularlight source with a unique color; controlling the particular lightsource with a unique lighting effect; or controlling the particularlight source with a unique light intensity.

In other examples, visually identifying the set of surgical items maycomprise presenting first mixed reality information in a visualizationdevice, and in this case, visually distinguishing the particularsurgical item may comprise presenting second mixed reality informationin the visualization device. In this example, visually distinguishingthe particular surgical item may comprise one or more of presenting thesecond mixed reality information with a unique color, presenting thesecond mixed reality information with a unique effect, or presenting thesecond mixed reality information with a unique intensity. Other visualeffects, as described above, may also be used. More generally, however,identifying a set of surgical items and visually distinguishing theparticular surgical item may comprise presenting mixed realityinformation.

In some examples, the mixed reality information presented on surgicalitems by visualization device 213 distinguishes a first subset of thesurgical items, a second subset of the surgical items and a third subsetof the surgical items, wherein the first subset of the surgical itemscorresponds to already used surgical items, the second subset of thesurgical items corresponds a currently used surgical item and the thirdsubset of the surgical items corresponds to subsequently needed surgicalitems. As described herein, the set of the surgical items may comprisesounders for an orthopedic shoulder repair surgery and the surgical itemcomprises a current sounder to be used.

In some examples, the set of surgical items comprise sounders for anorthopedic shoulder repair surgery and one or more punching tools for anorthopedic shoulder repair surgery. In some examples, the set ofsurgical items may comprise sounders for an orthopedic shoulder repairsurgery, one or more punching tools for an orthopedic shoulder repairsurgery, and one or more compacting tools for an orthopedic shoulderrepair surgery. In some examples, the set of the surgical items maycomprise sounders for an orthopedic shoulder repair surgery, one or morepunching tools for an orthopedic shoulder repair surgery, and one ormore compacting tools for an orthopedic shoulder repair surgery, and oneor more planing tools for an orthopedic shoulder repair surgery.

In still other examples, visualization device 213 may present virtualelements or surgical items may be identified by integrated lighting asdescribed herein on an item-by-item basis, without necessarilyilluminating the entire set. For example, visualization device 213 maypresent virtual elements sequentially with a surgical process in orderto sequentially identify each surgical item (item-by-item) in theprocedure. Or integrated lighting may be controlled by an externalprocessing device so as to sequentially illuminate each surgical itemaccording to a surgical process, in order to sequentially identify eachsurgical item (item-by-item) in the procedure.

FIG. 103 is a flow diagram illustrating a method for identifyingsurgical items in a surgical procedure and changing the identificationof such surgical items based on the stage of the surgical procedure. Themethod of FIG. 103 may be performed, for example, with a visualizationdevice 213 that presents virtual elements or with the use of surgicalitems that include integrated lighting as described herein. As shown inFIG. 103, a method includes visually identifying a first subset ofsurgical items as being previously used in the surgical procedure(10300), and visually identifying a second subset of surgical items asbeing in current use for the surgical procedure (10302). The method,e.g., as performed by either visualization device 213 with the use of MRor by surgical items with integrated lighting, may further includevisually identifying a third subset of surgical items as being neededfor subsequent use in the surgical procedure (10304).

According to the example of FIG. 103, at the next step of the medicalsurgical (“yes” branch of 10306), the “current surgical item” is movedfrom the second subset to the first subset and the next “subsequentsurgical item” is moved from the third subset to the second subset(10308). For example, one or more processors of visualization device 213may track the procedure based on feedback and/or virtual icon selectionby the nurse, and such feedback or icon selection can causevisualization device 213 to move the assignment of surgical items fromone set to another. Or an external processor that controls integratedlighting of the surgical items may be configured to track the procedurebased on feedback by the nurse, and such feedback can cause the externalprocessor to move the assignment of surgical items from one set toanother.

In this way, when the stage of the surgical procedure changes, the itemidentification (i.e., lighting or the presentation of virtual elementsfor item identification) can likewise change so as to identify newsurgical items (i.e., the next “subsequent surgical item”) to the nursefor current use. The manner of visually identifying the surgical itemsmay correspond to any of the examples described herein, including theuse of lights within the surgical items, the use of mixed realityinformation presented by visualization device 213, or both.

Various examples have been described. For example, item identificationand tracking techniques have been described for a surgical procedurethat leverage surgical items that are designed with advanced features,such as sensors, accelerometers, and light sources. Moreover, mixedreality techniques have also been described for use in item tracking anditem identification during a surgical procedure. Techniques forrecording and documenting the surgical procedure have also beendescribed, which may use automation.

The item identification techniques of this disclosure have beendescribed by using lights on the surgical items (e.g., tools)themselves, which may be controlled by an external device according to apreoperative plan. Alternatively or additionally, item identificationtechniques of this disclosure have been described by using mixedreality, e.g., by presenting one or more virtual elements on avisualization device 213 to identify surgical items needed in thesurgical procedure, which may be defined according to a preoperativeplan. In some situations, however, a surgeon may deviate from thepreoperative plan, and in these cases, the techniques of this disclosuremay adapt or adjust the item identification process according to thechanges made by the surgeon. Moreover, surgical item selection and/orsurgical item use may be used to identify such a deviation from thepreoperative plan. That is to say, in some examples, a current surgicalitem in use may be used to identity a next surgical item likely to beused. In other words, processing device 8304 may be configured toselect, based on detection of a current surgical item in use, a nextsurgical item and present virtual information that identifies the nextsurgical item. For example, if compactor size 3 is being used, thisinformation may be used by the system to recommend compactor size 4 asthe next surgical item, or possibly to recommend punching tool size 3 tobe used, or possibly to highlight either compactor size 4 or punchingtool size 3 to be two alternative surgical items that the surgeon mayneed next. The current surgical item being used in the surgicalprocedure may be highly indicative of the next surgical item likely tobe used, and processing device 8304 of the example shown in FIG. 83 orvisualization device 213 in the example shown in FIG. 84 may beconfigured to identify a likely next surgical item (or a few possiblelikely surgical items) based on the current surgical item in use.

Furthermore, when the surgical procedure deviates from a preoperativesurgical plan in the operating room, processing device 8304 of theexample shown in FIG. 83 or visualization device 213 in the exampleshown in FIG. 84 may change or adapt the item identification process soas to identify different surgical items that are likely to be needed inthe surgical process, given that the surgical process has been changedby the physician in the operating room.

For example, processing device 8304 of the example shown in FIG. 83 orvisualization device 213 in the example shown in FIG. 84 may store aninformation model to track the step being performing (e.g., byvisualization or interaction of surgeon with virtual guidance) andpredicting which surgical step is next. Then, processing device 8304 ofthe example shown in FIG. 83 or visualization device 213 in the exampleshown in FIG. 84 may identify the next surgical item to be used based onthis prediction. The information model may include a decision tree thataccounts for branches from steps (e.g., step 4 becomes step 4 a or step4 b depending on which happened in step 3). The surgical items for thebranches may be different, and surgical items may be identified bylighting on the surgical items or by virtual information presented on oraround such surgical items according to which branch is predicted. Inthis way, use of the virtual guidance system or lighting (and associatedinformation model and prediction) can provide an item tracking methodthat is more adaptive to changes in the operating procedure in realtime. Moreover, the item tracking may be synchronized with virtualguidance described herein, e.g., providing item prediction based onvirtual guidance steps. Surgical item predictions for different virtualguidance steps can be stored in a memory of visualization device 213 (orin a memory of processing device 8304).

In some examples, the information model that defines item prediction anditem identification can be defined based on camera or other sensorinput, e.g., possibly using machine learning to learn and predict wherea surgeon is within the given surgical procedure (e.g., which step) inorder to predict the next step (by identifying the current step of asurgical plan) and to predict the next surgical item that is needed(based on the current step or the current surgical item being used). Theinformation model may comprise a sequence of steps or may include moreadvanced information, in that it could describe different scenariosbased on different events occurring during surgery. In some examples,the information model may define a decision tree-structure that helps topredict the next step of the procedure (by identifying the current step)and/or to help predict the next surgical item that is needed (based onthe current step or the current surgical item being used). The decisiontree can be stored in a memory of visualization device 213 (or in amemory of processing device 8304), and based on this decision tree,visualization device 213 may present predictions of the next surgicalstep or visualization device 213 or processing device 8304 may presentor identify predictions of the next (or a set of possible next) surgicalitem(s) needed for the surgical procedure.

In some examples, processing device 8304 may select, based on thedetected change in the surgical procedure and based on machine learning,the next surgical item and present second virtual information thatidentifies the next surgical item. For example, processing device 8304(or visualization device 213) may apply machine learning over manysurgical procedures and may use decisions and changes made in pastsurgical procedures to drive machine learning and thereby predict thenext surgical item. In some cases, the machine learning may be based onpast surgical procedures, decisions and changes made by a specificsurgeon. In other words, the past surgical procedures, decisions andchanges made by a specific surgeon may be stored in memory and used asdata for driving a machine learning algorithm that can help to predictthe next surgical item. In this way, machine learning can help topredict the next surgical item needed for a given procedure, and thepredictions may be based on historical data, such as past surgicalprocedures, decisions and changes made by a specific surgeon. In somecases, historical data of a variety of surgeons may be used for thismachine learning, e.g., whereby past surgical procedures, decisions andchanges made by many surgeons can be used to drive machine learningalgorithm that can help to predict the next surgical item. For example,if a particular change is made to a surgical plan in the operating roomand if a similar change was made to the surgical plan for one or moreprevious surgeries, the surgical items used in subsequent steps of theprevious procedures (following the change made by the surgeon) may begood predictions of surgical items needed in similar subsequent steps ofthe current surgical procedure. By implementing a machine learningalgorithm, processing device 8304 (or visualization device 213) may beconfigured to predict surgical items needed for a procedure following aninteroperative change to that procedure.

As one example, a decision tree may be defined for humerus preparation,providing different branches for virtual guidance and surgical itemselection for anatomical preparation relative to reverse anatomicalpreparation. If the surgeon changes the plan in the operating room,e.g., changing from a preoperative surgical plan for an anatomicalrepair to a reverse anatomical repair, visualization device 213 mayadjust its predictions of the next surgical step or visualization device213 or processing device 8304 may adjust its predictions of the next (ora set of possible next) surgical item(s) needed for the surgicalprocedure.

As another example, visualization device 213 and advanced surgical itemswith sensors may document actual surgical parameters based on what wasactually performed by a surgeon, and predictions of virtual guidanceand/or surgical item selection may change from that defined in apreoperative surgical plan to that actually being performed in theoperating room.

As yet another example, item prediction may be linked to surgical itemsizes being used. Thus, if a particular size was used in previous step,the next surgical item may need to be selected to be a differentsurgical item of the same size or a similar surgical item of a differentsize. The surgical item prediction may be based on the surgical itemcurrently in use, and possibly the size of the current surgical item inuse. Moreover, if surgical items used or surgical item sizes change inthe operating room relative to the surgical items and sizes defined in apreoperative plan, visualization device 213 may adjust its predictionsof the next surgical step or visualization device 213 or processingdevice 8304 may adjust its predictions of the next (or a set of possiblenext) surgical item(s) needed for the surgical procedure. These andother examples of virtual guidance and virtual presentations onvisualization device 213 may be implemented via a software system likethe BLUEPRINT™ system available from Wright Medical.

Specific sets of surgical items have been described and illustrated forexample purposes. However, the techniques may be useful for a widevariety of surgical procedures and a wide variety of surgical items. Thetechniques may be particularly useful for orthopedic surgicalprocedures, such as shoulder surgeries, ankle surgeries, knee surgeries,hip surgeries, wrist surgeries, hand or finger surgeries, foot or toesurgeries, or any joint repair surgical procedure or augmentation.Although the techniques may be useful in a wide variety of orthopedicprocedures, they may be especially useful in both anatomical andreverse-anatomical shoulder reconstruction surgeries. Indeed, thetechniques may be helpful for reversed arthroplasty, augmented reversearthroplasty, standard total shoulder arthroplasty, augmented totalshoulder arthroplasty, hemispherical should surgery, or other types ofshoulder surgery. Although FIGS. 86-108 discuss aspects of specificshoulder surgery steps, the techniques of this disclosure may apply toother types of shoulder surgeries. Moreover, ankle surgery may involvesthe use of multiple tools and jigs for drilling, cutting, preparingimplant cites and installing implants on a patients talus and tibia, andthe tool tracking techniques of this disclosure may utilized for anklesurgeries, especially when a variety of different tools are used in theprocedure. These and other examples are within the scope of thisdisclosure.

Patients who have damage in a joint frequently have limited range ofmotion in an appendage associated with the joint. For example, a patientwith a damaged left shoulder typically cannot move his or her left armthroughout a range of angles and positions typical of people withundamaged shoulders. In another example, a patient with a damaged anklemay be incapable of elevating or dropping his or her foot beyond aparticular angle relative to the patient's lower leg. Such damage may becaused by a variety of conditions and events, such as arthritis, sportsinjuries, trauma, and so on.

During preoperative phase 302 (FIG. 3), a healthcare professional mayevaluate the range of motion of an appendage associated with a joint aspart of evaluating the patient's condition. For instance, the healthcareprofessional may be able to determine the severity of damage or type ofdamage to a patient's joint based on the range of motion of an appendageassociated with the joint. Furthermore, the healthcare professional mayask the patient to identify points in the range of motion of theappendage at which the patient experiences pain in the joint associatedwith the appendage.

After a patient has undergone a surgery on a joint associated with anappendage (i.e., during postoperative phase 308), a healthcareprofessional may wish to evaluate the range of motion of the appendage.For instance, the appendage typically has limited mobility immediatelyafter the surgery, but the range of motion of the appendage shouldincrease as the patient heals. The patient's appendage failing toachieve expected ranges of motion within particular time windows may bea sign that additional interventions or physical therapy may benecessary. Accordingly, to check whether additional interventions orphysical therapy are necessary, the patient typically visits thehealthcare professional who may then perform a physical examination ofthe appendage's range of motion.

In some instances, to help increase the chances that the appendageachieves a full range of motion, the patient may undergo physicaltherapy that requires the patient to move the appendage in particularways. For example, a physical therapist may request the patient to movethe appendage through a particular range of motion. In some examples,during the postoperative phase, the patient may be requested to performphysical therapy exercises at home outside of the presence of a physicaltherapist or other healthcare professional. For instance, in an examplewhere a patient has undergone a shoulder arthroplasty on the patient'sleft shoulder, the patient may be prescribed a physical therapy exercisethat involves the patient attempting to move their left arm to aparticular angle relative to a sagittal plane, frontal plane, ortransverse plane or the patient.

A variety of challenges confront healthcare professionals and patientsin evaluating the range of motion of an appendage associated with ajoint. For example, during preoperative phase 302 and postoperativephase 308 (FIG. 3), it may be difficult for a healthcare professional toaccurately measure and assess the range of motion of the appendage.Additionally, it may be difficult for the healthcare professional toaccurately record positions of the appendage at which the patientexperiences pain in the joint associated with the appendage. Similarly,it may be difficult for a patient to know whether he or she is movingtheir appendages through prescribed ranges of motion. Moreover, it maybe inconvenient or costly for a patient to make frequent office visitsfor physical therapy or physical examination by a healthcareprofessional. However, avoiding such office visits through interactivetelemedicine sessions or remote monitoring may be challenging because itmay be difficult for patients to accurately move their appendages toprescribed angles without trained physical assistance, difficult forpatients to describe their ranges of motion accurately, and difficultfor patients to describe the positions of their appendages at which thepatients experience pain. Additionally, it may be challenging for ahealthcare professional to interpret and validate the information onrange of motion and pain provided by the patient. For instance, it maybe difficult for the healthcare professional to know from informationprovided by the patient whether the information provided by the patientis wrong or whether there is an actual need for intervention.

This disclosure describes techniques that may address one or more ofthese challenges. For instance, in accordance with an example of thisdisclosure, a computing system may obtain motion data describing amovement of a motion tracking device connected to an appendage of apatient. In this example, the computing system may determine, based onthe motion data, a range of motion of the appendage. Additionally, inthis example, the computing system may generate an extended realityvisualization of the range of motion of the appendage superimposed onthe patient or an avatar of the patient. By viewing the extended realityvisualization of the range of motion of the appendage, a user separatefrom the patient, or in addition to the patient, may be able tovisualize and describe the range of motion of the appendage. Theextended reality visualization of the range of motion may be a mixedreality (MR) visualization or a virtual reality (VR) visualization.

FIG. 109 is block diagram illustrating an example system 10900 forgenerating an extended reality visualization of a range of motion of anappendage of a patient, in accordance with a technique of thisdisclosure. As shown in the example of FIG. 109, system 10900 includes acomputing system 10902, a set of one or more extended reality (XR)visualization devices 10904A through 10904N (collectively, “XRvisualization devices 10904”), and a motion tracking device 10906. Inother examples, system 10900 may include more, fewer, or differentdevices and systems. In some examples, computing system 10902, XRvisualization devices 10904, and motion tracking device 10906 maycommunicate via one or more communication networks, such as theInternet. In some examples, motion tracking device 10906 may communicatewith computing system and/or one or more XR visualization devices 10904via a direct wireless communication link.

Computing system 10902 may include various types of computing devices,such as server computers, personal computers, smartphones, laptopcomputers, and other types of computing devices. In the example of FIG.109, computing system 10902 includes one or more processing circuits10908, a data storage system 10910, and a set of one or morecommunication interfaces 10912A through 10912N (collectively,“communication interfaces 10912”). Data store 10910 is configured tostore data, such as motion data. Communication interfaces 10912 mayenable computing system 10902 to communicate (e.g., wirelessly or usingwires) to other computing systems and devices, such as XR visualizationdevices 10904 and motion tracking device 10906. For ease of explanation,this disclosure may describe actions performed by processing circuits10908, data store 10910, and communication interfaces 10912 as beingperformed by computing system 10902 as a whole.

Various computing systems of orthopedic surgical system 100 (FIG. 1) mayinclude computing system 10902. For example, virtual planning system102, pre- and postoperative monitoring system 112, and/or anothersubsystem of orthopedic surgical system 100 may include computing system10902. In some examples, one or more of XR visualization devices 10904includes one or more components of computing system 10902. For instance,one or more of XR visualization devices 10904 may include one or more ofprocessing circuits 10908 of computing system 10902. Thus, in someexamples, some or all of the actions described in this disclosure asbeing performed by computing system 10902 may be performed by processingcircuits in one or more of XR visualization devices 10904. In someexamples, XR visualization devices 10904 include MR visualizationdevices, such as MR visualization device 213 (FIG. 2). In some examples,XR visualization devices 10904 include VR visualization devices.

Motion tracking device 10906 is a device configured to detect movementof an appendage of the patient. For instance, in some examples, motiontracking device 10906 may include a device that is connected to theappendage of the patient and detects movement of motion tracking device10906. Motion tracking device 10906 may, for example, be a device havingan inertial measurement unit (IMU) that tracks acceleration of motiontracking device 10906 in multiple dimensions (e.g., 3 dimensions). Insome examples, the IMU may also track an orientation of motion trackingdevice 10906 (e.g., with respect to a gravitational vector or a magneticpole). Motion tracking device 10906 may be or may include various typesof devices. For example, motion tracking device 10906 may be or mayinclude a smartwatch, a smartphone, a ring, a bracelet, an anklet, ahead-mounted device, eyewear, a special-purpose motion tracking device,or another type of device configured to detect movement of the device.

In examples where motion tracking device 10906 is or includes a devicethat is connected to the appendage of the patient and detects movementof motion tracking device 10906, motion tracking device 10906 may beconnected to the appendage of the patient in various ways. For instance,motion tracking device 10906 may be connected to a wrist, ankle, thigh,foot, toe, finger, head, knee, calf, upper arm, hand, jaw, or other bodypart of the patient. Motion tracking device 10906 may be connected tothe appendage of the patient in various ways. For example, motiontracking device 10906 may be held by the patient (e.g., as may be thecase when the patient holds motion tracking device 10906 in one of thepatient's hands); may be strapped to the patient (e.g., as may be thecase when motion tracking device 10906 is worn on the patient's wrist orankle); may be attached with adhesive, may rest on the patient due togravity and/or compression (e.g., as may be the case when motiontracking device 10906 includes eyewear or headwear); may be held inplace by compression (e.g., as may be the case when motion trackingdevice 10906 is worn as a ring or clamp; or in may connected to theappendage of the patient in other ways such that motion tracking device10906 moves with the appendage of the patient. In some examples, motiontracking device 10906 may instruct the patient to start a movement at acalibration position (e.g., arm straight down) and track movementsrelative to the calibration position.

In some examples, motion tracking device 10906 may include one or morecameras or other devices that visually detect the movement of theappendage. For instance, in some examples, one or more cameras may beintegrated into an XR visualization device worn by the patient. In someexamples where the one or more cameras are integrated into an XRvisualization device worn by the patient, the patient may need to bepositioned in front of a mirror so that the camera is able to captureimages of the movement of the appendage of the patient.

In accordance with an example of this disclosure, computing system 10902may obtain motion data describing a movement of an appendage of apatient. For example, computing system 10902 may obtain motion data thatcomprise IMU signals generated by an IMU of motion tracking device 10906during the movement of the appendage of the patient. In some examples,computing system 10902 may obtain video data that show the movement ofthe appendage of the patient. In some examples, a storage system (e.g.,storage system 206, memory 215, etc.) may store the motion data.

Computing system 10902 may determine, based on the motion data, a rangeof motion of the appendage. For example, computing system 10902 maydetermine based on the IMU signals how far motion tracking device 10906traveled during the motion of the appendage. In this example, based on apreviously determined distance of motion tracking device 10906 from thejoint associated with the appendage, computing system 10906 maydetermine the range of motion of the appendage as:

${range}\mspace{14mu} {of}\mspace{14mu} {motion}{= \frac{360*l}{2\pi r}}$

In the equation above, l is the distance motion tracking device 10906traveled and during the motion of the appendage and r is the distance ofmotion tracking device 10906 from the joint associated with theappendage. In an example where computing system 10902 obtains video datathat show the movement of the appendage of the patient, computing system10902 may apply image analysis to video data to identify a major axis ofthe appendage and, in some examples, an applicable axis (e.g., a frontalaxis, transverse axis, sagittal axis, etc.) of the patient. For example,computing system 10902 may apply a neural network (e.g., convolutionalneural network) trained to recognize areas within images that containthe appendage. In this example, computing system 10902 may thendetermine a longest dimension of the recognized areas as the major axisof the appendage. In some examples, computing system 10902 may receivean indication of user input indicating the applicable axis. In someexamples, computing system 10902 may apply a neural network (e.g., aconvolutional neural network) to determine the applicable axis. In suchexamples, computing system 10902 may compare these axes to determineangles defining the range of motion.

The ranges of different motions of an appendage may be significant. Forinstance, FIG. 110 is a conceptual diagram illustrating example motionsof a patient's right arm that occur in the patient's shoulder. As shownin FIG. 110, flexion occurs when the patient raises the patient's armupward in a sagittal plane and extension occurs when the patient lowersthat patient's arm downward in the sagittal plane. Hyperextension occurswhen the patient moves the patient's arm in the sagittal plane past afrontal plane that runs through the patient's shoulder joint.Furthermore, as shown in FIG. 110, abduction is raising the patient'sarm in a frontal plane away from the center of the patient's body.Adduction is moving the patient's arm in the frontal plane toward thecenter of the patient's body. As shown in FIG. 110, internalrotation/inward rotation and external/outward rotation occurs when thepatient's arm rotates at the shoulder.

In some examples, the patient may be instructed (e.g., by motiontracking device 10906) to perform a type of movement and the patient mayperform a movement in response to being prompted to perform the type ofmovement. For instance, the type of movement may be a movement of theappendage in a plane that passes through a joint associated with theappendage. As an illustration, for shoulder range of motion, the patientmay be instructed (e.g., by motion tracking device 10906), in a firstexercise, to move the arm from a point of flexion to a point ofextension and, if possible, to a point of hyperextension; in a secondexercise, to move the arm from a point of abduction to a point ofadduction, and; in a third exercise, to move the arm from a point ofexternal/outward rotation to a point of internal/inward rotation. Inanother example, for ankle range of motion, the patient may beinstructed (e.g., by motion tracking device 10906), in one exercise, tomove the patient's foot from a point of plantarflexion to a point ofdorsiflexion.

During each exercise, motion tracking device 10906 may track thesepoints to form a representation of the patient's range of motion. Forexample, upon instructing the patient to undertake movement fromabduction to adduction, or vice versa, motion tracking device 10906 mayrecord data indicating for maximum points of abduction and adduction forreview by an orthopedic surgeon, physical therapist or other user. Inanother example, upon instructing the patient to undertake movement fromplantarflexion to dorsiflexion, motion tracking device 10906 may recorddata indicating maximum points of plantarflexion and dorsiflexion forreview. For instance, motion tracking device 10906 may record angles formaximum points relative to particular planes, polar coordinates of themaximum points, spherical coordinates of the maximum points, Cartesiancoordinates of the maximum points, or other types of data to indicatethe maximum points. For instance, in one example, based on the distancethat motion tracking device 10906 travels and the distance of motiontracking device 10906 from a joint associated with the appendage, motiontracking device 10906 may determine an angle relative to an applicableplane (e.g., using the equation above) and thereby determine a pair ofpolar coordinates defined by the distance of motion tracking device10906 from the joint associated with the appendage and the angle. Motiontracking device 10906, similarly, may record relative or absolutecoordinates for maximum points of flexion and extension, and relative orabsolute coordinates for maximum points of external/outward rotation andinternal/inward rotation during corresponding exercises performed by thepatient at the instruction of motion tracking device 10906 for review byan orthopedic surgeon, physical therapist or other user.

The patient may be instructed (e.g., by motion tracking device 10906) toperform a type of movement that requires articulation of a jointassociated with an appending in multiple dimensions. For instance, thepatient may be instructed to perform movements related to dailyactivities, such as moving the hand to a contralateral position, movingthe arm to comb the patient's hair (or otherwise bring the patient'shand to the patient's head), or place the hand on the patient's backpocket (or buttock) on the side of the patient's body adjacent thepertinent arm undertaking the movement. For each of these commonmovements, motion tracking device 10906 may record relative or absolutecoordinates associated with each of these movements for review by anorthopedic surgeon, physical therapist or other user. Range of motionassociated with similar common movements for other joints or appendagesor other body parts of the patient may be evaluated.

Additionally, computing system 10902 may generate an XR visualization(e.g., a MR, AR, or VR visualization) of the range of motion of theappendage. XR visualization devices 10904 may output the XRvisualization of the range of motion of the appendage for display to auser. XR visualization devices 10904 may output the XR visualization ofthe range of motion of the appendage such that the XR visualization ofthe range of motion of the appendage is superimposed on an image of thepatient or an avatar of the patient. In some examples, the image of thepatient is an image formed by light reflecting, directly or indirectly,off the patient. The image formed by light reflecting indirectly off thepatient may be reflected by a mirror after being reflected off thepatient and prior to detection by one or more cameras of one or more ofXR visualization devices 10904. In some examples, the image of thepatient is a previously captured 2-dimensional or 3-dimensional image ofthe patient. The avatar of the patient may be a virtual human figurerepresenting the patient.

In some examples, the XR visualization of the range of motion issuperimposed on the image of the patient or the avatar of the patientsuch that some visible portion of the XR visualization of the range ofmotion appears to the user to overlap the image of the patient or theavatar of the patient. In some examples, the XR visualization of therange of motion is superimposed on the image of the patient or theavatar of the patient in the sense that the XR visualization of therange of motion is superimposed on a scene that contains the image ofthe patient or the avatar of the patient, regardless of whether anyvisible portion of the XR visualization appears to the user to overlapwith the image of the patient or the avatar of the patient.

The extended reality visualization of the range of motion of theappendage may have various forms. For instance, in one example, the XRvisualization of the range of motion of the appendage may include avirtual arc spanning an angle between furthest points in the range ofmotion of the appendage. In some examples, a focal point of the virtualarc may be located at a joint associated with the appendage. In thisexample, if the appendage is the patient's left arm, the virtual arc mayappear superimposed in an area of space around the patient's leftshoulder and a focal point of the virtual arc may be located at thepatient's left shoulder joint. The ends of the virtual arc maycorrespond to the limits to which the patient is able to move thepatient's left arm in a particular plane of the patient's body, such asa frontal plane (e.g., during abduction and adduction), sagittal plane(e.g., during flexion and extension or during internal and externalrotation), or other plane of the patient's body.

In some examples where the XR visualization of the range of motion ofthe appendage includes such a virtual arc, the virtual arc may beassociated with a virtual protractor with labeled angles. In some suchexamples, the labeled angles of the virtual protractor may be relativeto a plane or axis of the patient's body. For instance, if the appendageis the patient's left arm and the range of motion is in the patient'sfrontal plane (e.g., during abduction and adduction), the angles may berelative to a frontal axis that passes through the patient's leftshoulder joint. If the appendage is the patient's left arm and the rangeof motion is in the patient's sagittal plane (e.g., during flexion andextension), the angles may be relative to a longitudinal axis thatpasses through the patient's left shoulder joint.

The labeled angles of the virtual protector may include angles at theends of the virtual arc. In some examples, the labeled angles of thevirtual protractor may include intermediate angles between the angles atthe ends of the virtual arc. In some examples, the labeled angles may bevisible in the XR visualization, but the virtual arc itself is invisiblein the XR visualization. In some examples, the virtual protractor isincorporated into the virtual arc or separate from the virtual arc. Insome examples where there are multiple virtual arcs, there may be asingle virtual protractor for all of the virtual arcs or there may beseparate virtual protractors for two or more of the virtual arcs.

Furthermore, in some examples where the XR visualization of the range ofmotion of the appendage includes a virtual arc, one or more XRvisualization devices 10904 may output a current line that is radialfrom a focal point of the virtual arc and in a plane of the virtual arc.In such examples, computing system 10902 updates the current line sothat the current line remains aligned in real time with a major axis ofthe appendage as the patient moves the appendage in the plane of thevirtual arc. In this way, the patient may be able to better visualizethe current angle of the appendage relative to the virtual arc.Computing system 10902 may use the motion data, such as IMU signalsand/or video data, to determine the current position of the appendagewhen generating the XR visualization of the current line. In someexamples, computing system 10902 may record positions of the currentline as the patient moves the appendage.

XR visualization devices 10904 may present XR visualizations of therange of motion to various users. For instance, in some examples, one ofXR visualization devices 10904 may present the XR visualization of therange of motion to the patient. For example, the patient may wear orotherwise use an XR visualization device, such as an MR, AR, or VRvisualization device. In the case where the XR visualization device usedby the patient is an MR visualization device, the MR visualizationdevice may be visualization device 213. In this example, the XRvisualization device presents the XR visualization of the range ofmotion of a joint associated with an appendage when the patient looks atthe joint while wearing or otherwise using the XR visualization device.In another example where the patient wears or otherwise uses an XRvisualization device, the patient may stand in front of a mirror. Inthis example, the XR visualization device may present the XRvisualization of the range of motion superimposed on the patient'sreflection or an avatar of the patient composed based on the patient'sreflection. This may be especially useful when it may be difficult forthe user to directly see a joint or a range of motion of a joint in aparticular direction, as may be the case for the patient's shoulders orneck.

In examples where an XR visualization device presents the XRvisualization to the patient, the extended reality visualization mayinclude a first virtual arc spanning an angle between furthest points inthe actual range of motion of the appendage achieved by the patient. Inthis example, the XR visualization may also include a second virtual arcspanning an angle between furthest points in a target range of motion ofthe appendage. For example, a previously recorded range of motion of thepatient's left arm in a frontal plane running through the patient's leftshoulder joint may range from −90° to 10°, relative to a frontal axisrunning through the patient's left shoulder joint.

Accordingly, in this example, the first virtual arc may span from −90°to 10°. In this example, −90° may correspond to the patient's left armhanging loose at the patient's side and 10° may correspond to thepatient's left arm being slightly above horizontal. However, as part ofthe patient's postoperative rehabilitation or preoperative evaluation,the patient may be prompted to try to move the patient's left arm in thefrontal plane through a range of −90° to 20°. Accordingly, in thisexample, the second arc may span from −90° to 20°. In some examples, thefirst virtual arc and the second virtual arc are differently colored. Insome examples, the first virtual arc is presented in the XRvisualization as a segment of the second virtual arc.

In some examples, a target range of motion of an appendage is a typicalrange of motion of a healthy individual. In some examples, a targetrange of motion of an appendage is a typical range of motion for apatient at a particular point in a postoperative recovery process. Insome examples, a target range of motion of an appendage is apatient-specific range of motion, which may be determined by ahealthcare professional or planning software, such as BLUEPRINT™ byWright Medical. Because the patient is able to perceive the secondvirtual arc in the XR visualization, the patient may be better able todetermine whether the patient is able to achieve the target range ofmotion than if the patient was merely told in writing or vocally thefurthest angles of the target range of motion.

As mentioned above, the target range of motion may be patient specific.Thus, in one example, there may be a first target range of motion thatis specific to a first patient. In this example, computing system 10902may obtain motion data describing a movement of the appendage of asecond patient. Furthermore, in this example, computing system 10902 maydetermine, based on the motion data, a range of motion of the appendageof the second patient. Computing system 10902 may also, in this example,generate, for display by an extended reality visualization device wornby the second patient, a second XR visualization of the range of motionof the appendage of the second patient superimposed on an image of thesecond patient or an avatar of the second patient. The second XRvisualization may include a virtual arc spanning an angle betweenfurthest points in a second target range of motion of the appendage ofthe second patient. The second target range of motion is different fromthe first target range of motion.

In some examples, the patient may be prompted to attempt to move theappendage to reach the furthest points in a target range of motion ofthe appendage, such as the target range of motion represented by thesecond virtual arc in the example above. The patient may be prompted toattempt to move the appendage to reach the furthest points in the targetrange in various ways. For example, computing system 10902 may cause oneor more speakers of an XR visualization device worn by the patient tooutput sound prompting the patient to move the appendage in the targetrange of motion. In some examples, computing system 10902 may cause awritten prompt to appear in the XR visualization presented to thepatient.

In some examples, the patient may be prompted as part of a preprogrammeddiagnostic or physical therapy session that does not actively involve ahealthcare professional. In some examples, a smartphone or other deviceused by the patient may prompt the patient to engage in physical therapyexercises or may specify the target range of motion. A smartphone,smartwatch or other device may also be the motion tracking device 10906used to record range of motion, hence serving the dual purposes ofissuing prompts or instructions for the patient to undertake range ormovement exercises and recording data representing the resultant motion.In particular, a smartphone may be held, or a smartwatch or otherwearable may be worn, by the patient during movement of an appendagethrough specified range of motion exercises.

The patient may be prompted to attempt to move the appendage to reachthe furthest points in the target range in response to various events.For example, during an interactive session involving the patient and ahealthcare professional, computing system 10902 may receive anindication of user input (e.g., voice command, mouse click, tap, etc.)from the healthcare professional to prompt the patient to attempt tomove the appendage to reach the furthest points in the target range ofthe appendage. In this example, computing system 10902 may prompt thepatient to attempt to move the appendage to reach the furthest points inthe target range in response to the indication of user input from thehealthcare professional. In some examples, during an interactivein-person or telemedicine session involving the patient and a healthcareprofessional, the healthcare provider may vocally prompt the patient.

The patient may receive the prompt from various devices. For example,motion tracking device 10906 may prompt the patient. For instance, in anexample where motion tracking device 10906 is a smartphone orsmartwatch, motion tracking device 10906 may display an on-screenmessage containing the prompt. In some examples, motion tracking device10906 may output generate audio that prompts the patient to perform thetype of movement of the appendage. Furthermore, in some examples, an XRvisualization device worn by the patient may prompt the patient. In somesuch examples, motion tracking device 10906 and/or XR visualizationdevices may generate such prompts in response to signals generated bycomputing system 10902.

The prompts received by the patient may include various types ofinformation. For example, a prompt may include text, video, and/or audiodescribing exercise that the patient is to perform. In examples where anXR visualization device worn by the patient presents the prompt, an MRor AR visualization may contain the prompt.

As noted above, one of the challenges confronting patients andhealthcare providers is that it may be difficult for the patient toprecisely express the points at which the patient experiences pain asthe patient moves an appendage through a range of motion. Knowing thepoints at which the patient experiences pain may help a healthcareprovider (and/or an AI-based diagnostic tool) learn about the patient'scondition. To address such challenges, computing system 10902 may, insome examples where the XR visualization is presented to the patient,receive indications of user input indicating one or more pain pointswithin the range of motion. The patient may experience pain when theappendage is at the pain points. Thus, the patient may provide theindication of user input when the appendage is at a point within therange of motion where the patient experiences pain. In response toreceiving the indication of the user input, computing system 10902 maygenerate, based on the motion data, data indicating a pain point, wherethe pain point is a position or positions of the appendage of thepatient at which the patient experiences pain.

For instance, if the appendage is the patient's left arm and computingsystem 10902 receives an indication of user input at a time when thepatient has raised the patient's left arm 10° above a frontal axisrunning through the patient's left shoulder joint, computing system10902 may determine that the patient has a pain point when the patient'sleft arm is 10° above the coronal axis. In this example, the patientdoes not need to specify that the pain occurs when the patient's leftarm is 10° above the frontal axis. Rather, the patient merely needs toprovide input when the patient feels pain and computing system 10902determines that the angle of the patient's left arm was 10° when thepatient felt the pain. Without the use of this technique, it may bedifficult for the patient to accurately say that the pain point occurswhen the patient's left arm is 10° above the frontal axis. Computingsystem 10902 may store (e.g., in data storage system 10910 of computingsystem 10902) data indicating the pain points.

Computing system 10902 may receive the indication of user input for apain point in one or more ways. For example, computing system 10902 mayreceive (e.g., via a microphone of an XR visualization device worn bythe patient, a microphone of motion tracking device 10906, or amicrophone of another device) a vocal indication from the patient whenthe patient moves the appendage to a point where the patient experiencespain. For instance, in this example, the patient may say “pain” when thepatient moves the appendage to a point where the patient experiencespain. In another example, computing system 10902 may receive (e.g., viaone or more cameras of an XR visualization device worn by the patient)video data showing the user performing a gesture (e.g., a hand gesture,head gesture, etc.) that indicates that the user has experienced pain atthe current point in the range of motion. In other examples, computingsystem 10902 may receive the indication of user input as a tappinggesture, a button push, or another form of user input.

In some examples, one or more of XR visualization devices 10904 maypresent the extended reality visualization of the range of motion to oneor more healthcare professionals, such as doctors, surgeons, or nurses.In other words, an XR visualization device may be worn by a healthcareprofessional. For example, a healthcare professional and the patient mayengage in an interactive session during which the healthcareprofessional wears or otherwise uses one or more XR visualizationdevices 10904. In this example, the healthcare professional and thepatient may be in separate locations. In other words, an XRvisualization device may present the extended reality visualization tothe healthcare professional during an interactive session with thepatient in which the healthcare professional and the patient are inseparate locations, as may be the case during a telemedicine session.

Alternatively, in this example, the healthcare professional and thepatient may be in the same location, such as during an office visit, andthe healthcare professional and patient may speak directly to oneanother. In other words, an XR visualization device may present the XRvisualization to the healthcare professional during an interactivesession with the patient in which the healthcare professional and thepatient are in the same location. In either case, the XR visualizationdevice may present the XR visualization of the range of motion to thehealthcare professional in real time so that the healthcare professionalis able to visualize the range of motion that the patient is able toattain.

In this example, the healthcare professional may communicate with thepatient (e.g., via the XR visualization device worn by the healthcareprofessional and a XR visualization device worn by the patient, via oneor more other communication devices, or directly in person) to instructthe patient to attempt to move the patient's appendage through varioustarget ranges of motion. In some examples, the XR visualizationpresented to the healthcare professional indicates one or more targetranges of motion.

In some examples where one of XR visualization devices 10904 presentsthe extended reality visualization to a healthcare provider, another oneof XR visualization devices 10904 may present another XR visualizationto the patient. For example, during a telemedicine session or during anoffice visit, both the healthcare professional and the patient may wearor otherwise use XR visualization devices. In this example, the XRvisualization devices may present XR visualizations to the healthcareprofessional and the patient showing one or more of an achieved range ofmotion of an appendage of the patient, a target range of motion, painpoints, etc.

In some examples where the XR visualization is presented to a healthcareprofessional, the healthcare professional views the XR visualizationoutside the context of an interactive session with the patient. In otherwords, an XR visualization device may present the XR visualization to ahealthcare professional during a session in which the patient is notinvolved. For example, the patient may perform various exercises thattest the ranges of motion of an appendage and computing system 10902 maystore (e.g., in data storage system 10910) data indicating the ranges ofmotion of the appendage.

In this example, computing system 10902 may use the stored data to latergenerate XR visualizations of the ranges of motion of the appendage. Inthis example, an XR visualization device worn by the healthcare providermay present the extended reality visualizations to the healthcareprovider at any time after the patient performs the exercises. In thisway, the healthcare provider may be able to effectively visualize orconceptualize the ranges of motion of the patient's appendage in a waythat may be difficult if the healthcare provider were merely looking atwritten notes indicating angles of the ranges of motion.

In some examples, computing system 10902 may send a notification to thehealthcare provider when computing system 10902 receive informationindicating that computing system 10902 has received new range of motiondata or that the patient is engaging in range of motion exercises. Insome examples, computing system 10902 may provide information to ahealthcare provider indicating whether the patient has engaged inassigned physical therapy exercises.

Furthermore, in some examples, computing system 10902 may presentinformation to a healthcare provider that shows the evolution of thepatient's range of motion over time. For example, computing system 10902may provide information to an XR visualization device that enables theXR visualization device to display virtual arcs corresponding to thatpatient's range of motion at various times. In some examples, the XRvisualization device may animate the virtual arcs to further helpdemonstrate the changes in the patient's range of motion. In someexamples, an XR visualization device worn by the patient may displaysimilar virtual arcs to help educate the patient with regard to theevolution of the patient's range of motion.

The XR visualizations presented to a healthcare professional may includevarious virtual objects. In some examples where the XR visualization ispresented to a healthcare professional, the XR visualization may includea virtual protractor, as described elsewhere in this disclosure.Furthermore, in some examples, one or more virtual arcs for the range ofmotion and target range of motion may be presented in the visualization,as described elsewhere in this disclosure. Thus, in one example, the XRvisualization presented to the healthcare provider may include a firstvirtual arc spanning an angle between furthest points of a range ofmotion of the appendage. In this example, the XR visualization furtheror alternatively includes a virtual arc spanning an angle betweenfurthest points in a target range of motion of the appendage.

In some examples, the XR visualization presented to a healthcareprofessional may include indications of the pain points as experiencedby the patient. That is, a pain point may be marked in the XRvisualization of the range of motion. For instance, in one example, theXR visualization presented to the healthcare professional may include avirtual arc corresponding to an achieved or target range of motion ofthe patient's appendage and may also include virtual indicators on thevirtual arc indicating points at which the patient experiences pain. Inother words, the XR visualization of the range of motion of theappendage may include a virtual arc spanning an angle between furthestpoints in the range of motion of the appendage and a pain point ismarked on the virtual arc.

Furthermore, irregularities in the motion of an appendage may provideuseful diagnostic information. For example, the patient may be able tolower the appendage in a controlled way through a first part of therange of motion but cannot easily control the motion of the appendage ina second part of the range of motion so that the appendage dropsquickly. In this example, this information may be diagnostic of aparticular type of health condition, such as tears in particular musclesor slippages in implanted joint replacement surfaces. In anotherexample, sudden or unusual accelerations or decelerations of the motionof an appendage at consistent points within a range of motion mayprovide valuable diagnostic information. In some instances, theseirregularities in the motion of an appendage may be imperceptible to thepatient. Such diagnostic information may be useful at various points inthe surgical lifecycle, such as during preoperative phase 302 andpostoperative phase 308.

Accordingly, computing system 10902 may determine, based on the motiondata, whether there are irregularities in the motion of an appendage ofa patient. For example, computing system 10902 may compare sets ofmotion data generated from multiple movements of the appendage throughthe range of motion to determine whether there is any consistent patternof motion. For instance, computing system 10902 may apply a dynamic timewarping algorithm to generate, based on set of motion signals fordifferent times the patient moved the appendage through the range ofmotion, a signal representative of the acceleration of the appendagethrough the range of motion. Computing system 10902 may then compare theresulting signal to a signal representative of acceleration of theappendage through the range of motion in a typical individual.

In some examples, an extended reality visualization of the range ofmotion may include information regarding irregularities in the motion ofthe appendage. Thus, computing system 10902 may determine, based on themotion data, a point in the range of motion of the appendage at which anirregular movement of the appendage occurs and may generate the extendedreality visualization of the range of motion such that the extendedreality visualization indicates the determined point in the range ofmotion at which the irregular movement of the appendage occurs.

For example, the XR visualization may include virtual points, virtualarcs, or other virtual objects indicating where there are irregularitiesin the motion of the appendage within the range of motion of theappendage. Information regarding irregularities in the motion of theappendage may be presented to the patient, a healthcare professional, oranother user of orthopedic surgical system 100. For instance, in anexample where the information regarding the irregularities in the motionof the appendage are presented to the patient, the patient may beprompted (e.g., by a XR visualization device worn by the patient, asmartphone used by the patient, motion tracking device 10906, etc.) totry to move the appendage so that the irregularity does not occur.

In this example, the irregularity may be confirmed if the patient cannotmove the appendage in the regular way despite being prompted to do so.In an example where the information regarding the irregularities in themotion of the appendage is presented to a healthcare professional (e.g.,by one of XR visualization devices 10904, a monitor, or othervisualization device), the XR visualization showing the informationregarding the irregularities in the motion of the appendage may help thehealthcare professional with diagnosis or physical therapy.

FIG. 111 is a conceptual diagram illustrating an example extendedreality visualization of a range of motion, in accordance with atechnique of this disclosure. The XR visualization shown in the exampleof FIG. 111 may be perceived by a healthcare provider wearing orotherwise using an XR visualization device (e.g., one of XRvisualization devices 10904). Alternatively, the XR visualization shownin the example of FIG. 111 may be perceived by a patient wearing orotherwise using an XR visualization device (e.g., one of XRvisualization devices 10904) when looking into a mirror.

The XR visualization of FIG. 111 includes an image or avatar 11100 ofthe patient or part of the patient. Additionally, the extended realityvisualization of FIG. 111 includes a virtual arc 11102 that spans anangle corresponding to an actual range of motion of the patient's leftarm in a frontal plane passing through the patient's left shoulder joint(e.g., when performing adduction and abduction). Additionally, in theexample of FIG. 111, the XR visualization includes a virtual arc 11104that spans an angle corresponding to a target range of motion of thepatient's left arm in the transverse plane.

In other examples, virtual arc 11102 and virtual arc 11104 may be inother planes. Marker 11106 may correspond to a pain point. Marker 11110may correspond to a point or region of irregular movement of thepatient's left arm. Additionally, in the example of FIG. 111, theextended reality visualization includes a current line 11108 that isradial from a focal point of virtual arc 11102 and virtual arc 11104 andin a plane of the virtual arcs. Computing system 10902 may updatecurrent line 11108 so that current line 11108 remains aligned with amajor axis of the patient's left arm.

FIG. 112A is a flowchart illustrating an example operation of system10900 for range of motion analysis and visualization, in accordance witha technique of this disclosure. In the example of FIG. 112A, computingsystem 10902 may obtain motion data describing a movement of anappendage of a patient (11200). For instance, computing device 10902 mayobtain the motion data from a motion tracking device, set of cameras, orcombination thereof, as described in this disclosure.

Furthermore, in the example of FIG. 112A, computing system 10902 maydetermine, based on the motion data, a range of motion of the appendage(11202). Computing system 10902 may determine the range of motion of theappendage in accordance with any one or combination of examples providedelsewhere in this disclosure.

Computing system 10902 may generate an XR visualization of the range ofmotion of the appendage superimposed on the patient or an avatar of thepatient (11204). Computing system 10902 may generate the XRvisualization in accordance with any one or combination of examplesprovided elsewhere in this disclosure.

FIG. 112B is a flowchart illustrating an example operation of system10900 in accordance with a technique of this disclosure. In the exampleof FIG. 112B, motion tracking device 10906 of a patient outputs a promptto attempt to move an appendage of the patient to reach furthest pointsin a target range of motion of the appendage (11240). For instance,motion tracking device 10906 may output an audio or video description ofhow to move the appendage. As discussed elsewhere in this disclosure,motion tracking device 10906 may output the prompt in response tovarious events, such as in response to the indication of user input fromthe healthcare professional.

Additionally, in the example of FIG. 112B, motion tracking device 10906may generate motion data describing a movement of the appendage of thepatient (11242). For instance, an IMU of motion tracking device 10906may generate the motion data describing the movement. Motion trackingdevice 10906 may generate, based on the motion data, data indicating anachieved range of motion of the appendage of the patient (11244). Forexample, motion tracking device 10906 may generate data indicatingcoordinates of the starting and stopping points of the achieved range ofmotion. In some examples, motion tracking device 10906 may send the dataindicating the achieved range of motion to a remote database of patientdata (e.g., data storage system 10910).

Furthermore, in some examples, motion tracking device 10906 may receivean indication of user input when the appendage of the patient is at apoint within the range of motion where the patient experiences pain. Insuch examples, in response to receiving the indication of the userinput, the motion tracking device may generate, based on the motiondata, data indicating a pain point. The pain point may be a position ofthe appendage of the patient at which the patient experiences the pain.

FIG. 113 is a conceptual diagram illustrating an example setting inwhich a set of users use MR systems for educational purposes. In theexample of FIG. 113, a surgeon may wear or otherwise use a visualizationdevice (e.g., visualization device 213) of a first MR system 11300(e.g., MR system 212. The visualization device of MR system 11300 maypresent MR educational content 11302 to a trainer. Furthermore, in theexample of FIG. 113, another surgeon may wear or otherwise use avisualization device of a second MR system 11304, a medical devicemanufacturer representative may wear or otherwise use a visualizationdevice of a third MR system 11306, and/or a nurse may wear or otherwiseuse a visualization device of a fourth MR system 11308. In one example,a medical device manufacturer representative may wear or otherwise usevisualization device of MR system 11306 while the surgeon wears orotherwise uses the visualization device of MR system 11300. In thisexample, the visualization devices of MR system 11300 and MR system11306 may present the same MR education content 11302 to the medicaldevice manufacturer representative and the surgeon. In this example, themedical device manufacturer may explain the use of a medical device tothe surgeon while the medical device manufacturer and the surgeon viewthe same MR preoperative content.

This disclosure describes a number of multi-user collaborationtechniques for an orthopedic medical procedure that makes use of mixedreality (MR). Although the techniques may be useful in a wide variety oforthopedic procedures, they may be especially useful in both anatomicaland reverse-anatomical shoulder reconstruction surgeries. Indeed, thetechniques may be helpful for reversed arthroplasty, augmented reversearthroplasty, standard total shoulder arthroplasty, augmented totalshoulder arthroplasty, hemispherical should surgery, or other types ofshoulder surgery. More generally, however, the techniques may finduseful application with any orthopedic medical procedure that involvesmultiple participants in the procedure.

Various view sharing techniques and controls are described for mixedreality devices used within the operating room, whereby a user may beable to view the mixed reality presentation (or a portion thereof) ofother users in the operating room. Users in the operating room, as wellas remote users, may include medical caregivers, such as, for example,one or more surgeons, one or more nurses, one of more medical supportpersonnel (such as technical support representatives of a manufacturerof medical device instruments, implants, equipment or supplies),anesthesiologists, and other operating room personnel.

In some examples, MR content associated with the views of other users(or possibly fixed cameras) may be presented as a window view within thepresentation of a user that wants to see a view from another user orperspective. The view of a user may be at least a portion of what theuser is able see from the user's own perspective. In some examples, theviews of other users may be selectively presented as the main view ofthe user that wants to see the view seen by another user or perspective.Hence, a user may view a main view and one or more additional views,e.g., in sub-windows, and select any of the views to be presented as themain view. For example, a physician performing an orthopedic surgery maybe presented with a mixed reality view from his or her perspective as amain view, but that user may be able to select and see the view of otherpersons (or fixed cameras) within the operating room, e.g., by selectingone of the sub-windows or other information associated with the views,based on person, view name, or other information. This can allow thephysician to view his or her own perspective as a main view but quicklygain other perspectives within the operating room by viewing thesub-windows and/or one of the sub-windows temporarily as the main view.In some examples, the views of others may be presented as windows on amain view of a user, such that the user is able to see his or her mainview while simultaneously watching another view of another person in awindow presented on the main view. A surgeon, for example, may accessthe view of a nurse for a different angle or perspective, whileperforming a surgical step, and the view of the nurse may be shown tothe surgeon in a window on the surgeon's main view.

Moreover, this disclosure also describes mixed reality presentationsthat can vary for different users in the operating room. For example,the mixed reality presentation for a nurse may be different from that ofa physician, since the nurse and the physician have different roles inthe operating room and mixed reality may be tuned on a user-by-userbasis to help each user with their role in the medical procedure. Otherparticipants may have other roles, and their mixed reality presentationmay be defined to accommodate such differing roles. As other examples,the mixed reality presentation for medical device representatives,assistants, different nurses, and different physicians may differ fromone another. When accessing the view of another person, that person'sview may include both real world objects (such as views of the patient'sbone or other anatomy), as well as virtual objects (such as virtualelements presented in the MR presentation). Thus, when a physicianaccesses or observes the view of a nurse, the physician may view bothreal-world objects viewed by the nurse, as well as virtual objectspresented to the nurse in the nurse's MR presentation.

In some examples, some mixed reality information (e.g., virtual elementsof the mixed reality presentation) may be common for all participants,while other mixed reality information may be specific and only presentedto specific participants based on their role in the medical procedure.This disclosure also contemplates the ability to enable or select mixedreality information from other users on an object-by-object basis, e.g.,allowing a physician to enable or disable a virtual element that is partof a nurse's MR presentation so as to show that same element on thephysician's MR presentation. Thus, view sharing may include the abilityto see the entire view of other users (including real observed objectsseen by other users and virtual elements presented to the other users),or the ability to enable or disable virtual objects shown to others onan object-by-object basis. The virtual objects that may be enabled ordisabled may comprise mixed reality guidance features, user interfaces,widgets, or any virtual information that is included in an MRpresentation.

In still other examples, this disclosure describes techniques that canfacilitate participation in a medical procedure by one or more remoteparticipants, possibly with the use of virtual reality. In differentexamples, the described system and techniques may facilitate active andpassive participants, user-specific mixed reality guidance that isdefined for different roles in the operating room, view sharing andcontrol of view sharing, and other useful features described in greaterdetail below. Active participants, for example, may have some level ofcontrol over virtual content, view sharing, and the MR presentation ofother views, while passive participants may have the ability to view anMR presentation and virtual content without any control over the MRpresentation and virtual content.

As mentioned elsewhere in this disclosure, multiple users cansimultaneously use MR systems. For example, the MR systems can support aspectator mode in which multiple users each have a visualization deviceso that the users can view the same information at the same time. Inthis way, one or more spectators can be active or passive participantsin a preoperative, intraoperative, or postoperative procedures.

The ability to include active and passive participants to a proceduremay be highly desirable. In some examples, active roles may be assignedor re-assigned during a medical procedure or medical encounter. In otherexamples, varying levels of active roles may be defined. Indeed,different MR experiences and features may be provided to differentpersons using MR systems. For example, a physician may be presented witha first MR interface and a nurse may be presented with a second MRinterface that is different from the first MR interface. For example,selectable widgets 524 (FIG. 5) of a user interface 522 shown in an MRscreen 520 of visualization device 213 may be defined for a physician,and different selectable widgets 524 may be defined for a nurse (orother user) that uses a different processing device(s) 210 (FIG. 2) thanthat used by the physician. Other variations between the MR experiencefor a first user and the MR experience of a second user may also bedefined based on the user. For example, a passive participant user maybe presented with viewable MR elements without any selectable controlwidgets, which may be desirable for students or other passive observersto the procedure. Such passive MR participants may benefit from viewingMR elements of the procedure but may be prevented from being able toselect or modify anything in the MR world.

FIG. 114 is a block diagram of a system 11400 that includes multiple MRdevices that communicate with one another. Each of the illustrated MRdevices may comprise visualization devices similar to visualizationdevice 213 (FIG. 2, FIG. 5) described herein. FIG. 114 shows four MRdevices (11402, 11406, 11408 and 11410) although more or fewer devicescould be used. Visualization devices of the MR devices (11402, 11406,11408 and 11410) may be worn by users in an operating room during anorthopedic medical procedure, e.g., surgery. As shown, MR device 1(11402) includes a view sharing window 11404. According to thisdisclosure, MR device 1 (11402) is configured to present one or more ofthe views of other MR devices on MR device 1 (11402). Although shown asa view sharing window (11404), the view sharing may be presented inother ways.

MR device 1 (11402) may be configured for a particular user, whereas MRdevice 2 (11406), MR device 3 (11408) and MR device 4 (11410) may beconfigured for other users. In this way user specific rolls may bedefined in the operating room, and mixed reality presentations may bedefined for each participant. Since the participants may have differentroles in the operating room, the MR presentations may be different fordifferent users. For this reason, view sharing may be desirable, e.g.,to allow the physician to see a nurse's view or a technician's view, soas to allow for more collaboration in the operating room. View sharingmay also be desirable to enable different angles of view in theoperating room or different perspectives. View sharing may occur betweendifferent MR devices, and possibly from one or more additional fixedcameras (not shown in FIG. 114). In some cases, one or more of MRdevices (11402, 11406, 11408, 11410) may be located remotely relative tothe surgical procedure. In this case, the remote participant may be ableto view the same virtual elements as other participants, but thereal-world views may differ due to the MR devices being used indifferent locations. View sharing may be especially useful in this typeof setting so as to allow the remote participant to see the views oflocal participants.

Different levels of active participation and passive participation mayalso be defined for different active participants to an orthopedicsurgical procedure, such as an ankle surgery procedure or a shouldersurgery procedure. For example, a physician may be given particularlevels of control of the displayed elements of an MR presentation duringa medical procedure, and nurses or assistants may be given differentlevels of control. Also, the MR interface may vary for different users.Accordingly, physicians may be presented with an MR interface for thephysician that is designed to provide virtual guidance to aid thephysician with the procedure (such as virtual guidance features to aidthe physician with the procedure), and nurses may be presented with adifferent MR interface designed to aid the nurse with particular tasksof the nurse (such as tool selection and tracking and documentation ofthe procedure). Other types of users may also be defined, such asmedical device personnel, additional physicians, anesthesiologists,physician assistants, nurse assistants, medical technicians, or otherusers. In some examples, a master role may be defined to provide overallcontrol over the process, and in some cases, the master role may bedelegated among users. For example, referring again to FIG. 114, MRdevice 1 (11402) may be assigned as the master device, and other usersassociated with MR devices 2-4 (11406, 11408 and 11410) may have lesscontrol than MR device 1 (11402). In some cases, however, the masterrole may be assigned or changed amongst the users.

In some examples, this disclosure describes a MR system (e.g., MR system11400) comprising a first MR device (e.g., MR device 1 11402) configuredto provide first medical information to a first user via a first MRpresentation, and a second MR device (e.g., MR device 11406) configuredto provide second medical information to a second user via a second MRpresentation. The first and second MR devices may comprise visualizationdevices as described elsewhere in this disclosure. In some examples, thefirst medical information is the same as the second medical information.In some examples, the first medical information and the second medicalinformation are different. The first MR device may be controllable topresent the second MR presentation to the first user. For example, thefirst MR device may be configured to present the second MR presentationto the first user as a viewable window within the first MR presentation.In another example, the first MR device may be configured to allow thefirst user to select and view either the first MR presentation or thesecond MR presentation. Each of the MR presentations may include viewsof real-world objects and one or more virtual elements.

In some examples, a MR system (e.g., MR system 11400) may comprise afirst MR device (e.g., visualization device 213) configured to providefirst medical information regarding an orthopedic medical procedure to aphysician via a first MR presentation wherein the first MR presentationincludes physician-specific information associated with the orthopedicmedical procedure. The MR system may further include a second MR device(e.g., another visualization device 213) configured to provide secondmedical information to a second user via a second MR presentationwherein the second MR presentation includes information that differsfrom the first MR presentation. In some examples, the first medicalinformation is the same as the second medical information. In someexamples, the first medical information and the second medicalinformation are different. The first MR device may be controllable topresent the second MR presentation to the physician. For example, thefirst MR device may be configured to present the second MR presentationto the first user as a viewable window within the first MR presentation.In some examples, the first MR device is configured to allow the firstuser to select the first MR presentation or the second MR presentation.The physician-specific information may include surgical guidanceinformation. Moreover, in some examples, the second MR presentation mayinclude nurse-specific information associated with the orthopedicmedical procedure. For example, the nurse-specific information mayinclude surgical item information. In other examples, the second MRpresentation may include technician-specific information associated withthe orthopedic medical procedure. For example, the technician-specificinformation may include registration guidance for registering one ormore virtual elements in the second MR presentation. For instance, theregistration guidance may include guidance for how to register a virtualelement (e.g., a virtual 3D model of a bone) with a real-world object,such as a bone.

In some examples, a method according to this disclosure may comprisepresenting first medical information to a first user via a first MRpresentation on a first visualization device and presenting secondmedical information to a second user via a second MR presentation on asecond visualization device. In this example, the first visualizationdevice may be controllable to present the second MR presentation to thefirst user. The second MR presentation may include views of real-worldobjects and one or more virtual elements.

In some examples, a method according to this disclosure may comprisepresenting first medical information to a first user via a first MRpresentation on a first visualization device, receiving second medicalinformation comprising a second MR presentation from a secondvisualization device, and controlling the first visualization device toselectively present the second MR presentation on the firstvisualization device. In some examples, the first medical information isthe same as the second medical information. In some examples, the firstmedical information and the second medical information are different.These and other methods described in this disclosure may be executed byone or more processors, in which case the techniques may be embodied inprocessor-executable instructions stored in a computer readable medium.The one or more processors may comprise one or more processors ofvisualization device 213 and the computer-readable storage medium maycomprise a memory of the visualization device 213.

Workflow management tools and checklists may also be used to provideadditional checks and controls on the surgical process. When multiple MRparticipants are involved with a surgery, for example, different personsmay be assigned for approval of different steps. One person may need toapprove a first medical procedure step before the system allows thesurgery to proceed to the next step, and in some cases, a differentperson may need to approve a second medical procedure step before thesystem allows the surgery to proceed to yet another step. In this way,checks and approvals of surgical steps may be distributed amongstdifferent participants (or observers) to the surgery. Workflowmanagement tools and checklists may be especially helpful for complexmulti-step surgical procedures, such as a shoulder arthroplasty, anankle arthroplasty, or any other type of orthopedic surgery thatrequires many different steps.

In some examples, physician may be presented with a first MR interfaceand a nurse may be presented with a second MR interface that isdifferent from the first MR interface. For example, referring again toFIG. 114, MR device 1 (11402) may present the first MR interface for aphysician, and MR device 2 (11406) may present the second MR interfacefor a nurse. Moreover, other users, such as assistants or medical devicetechnicians may be presented with still other types of MR interfaces. Inthis way, the MR interface presented to each user may be specificallydefined or tuned to aid that participant with their specific role in theprocedure. For example, MR-based intraoperative guidance features(cutting axes, drilling axes, reaming axes, virtual jigs, implantpositioning targets, or other intraoperative aids) for use in guidingoperative steps in a surgical procedure may be presented to the surgeonresponsible for such steps, but some of these intraoperative guidancefeatures may not be presented to other users, such as nurses, assistantsor medical device technicians. Moreover, menus or controls for suchintraoperative guidance features may be presented to a physician on theMR device for that physician, but such menus or controls may beeliminated from the views of other users on their MR device(s).Similarly, tool selection prompts, medical device inventory tracking,and medical procedure documentation features may be presented to thenurse (as well as menus and controls), but some or all of these featuresmay be eliminated from the views of the surgeon. Also, other features orcontrols may be presented to a medical device technician, such asfeatures for registering virtual elements into the MR presentation, butsuch technician-specific MR for that technician may be eliminated fromthe views of surgeons and nurses. In some cases, users may be allowed toselecting enable or disable virtual elements shown to other users. Forexample, nurse-specific virtual elements may be selectively enabled ordisabled by the surgeon, if the surgeon finds a need or desire to viewvirtual elements that are typically shown to the nurse and not to thephysician.

Some virtual elements may be viewable by all participants, while othervirtual elements or controls may be viewable by only selectparticipants. In this way, the MR systems can help to guide eachparticipant with tools and features that are tuned to the role of thatparticipant in the medical procedure. Moreover, eliminating features orcontrols that are not pertinent to the role of a given participant canimprove the MR experience and help to eliminate visual clutter in the MRexperience. For example, tool selection prompts may be highly desirableto aid a nurse in identifying tools and a succession or sequence of theplanned use of such tools in the procedure. But a surgeon may find toolselection prompts to be distracting, particularly while the surgeon isfocused on a patient operative tissue or bone site, and the physicianmay simply rely on the nurse to provide the tools in their proper orderfor the procedure, e.g., as requested by the surgeon and/or as indicatedby virtual guidance provided to the nurse regarding tool selection andsequence and overall surgical procedure workflow. For these and otherreasons, selectable widgets 524 of a user interface 522 shown in an MRscreen 520 of visualization device 213 may be defined for a physician,and different selectable widgets 524 may be defined for a nurse (orother user) so that each MR experience can be better suited to the rollof that participant in the procedure.

The users of MR devices, including active participants and spectators,may be physically present during the procedure, using MR visualizationdevices. Alternatively, one or more users or spectators may bephysically remote relative to the location of the procedure. When one ormore users or spectators are located remotely, they may participate inthe procedure or view the procedure via remote display screens such ason laptop computers, desktop computers, smart phones, or other types ofdisplay screens. In other examples, remote participants may view theprocedure via a mixed reality (MR) visualization device, or with virtualreality (VR) rather than MR. If MR is used for remote participants, theremote participant may be able to view the same virtual elements asother participants, but the real-world views may differ due to the MRdevices being used in different locations. View sharing may beespecially useful in this type of setting so as to allow the remoteparticipant to see the views of local participants.

In some examples, a system may comprise both VR devices and MR devices.In this case, the VR environment presented to the remote user maycomprise images and objects captured and presented by MR systems thatare used by local participants. In this way, the MR environment, whichincludes real-world objects, may be captured and presented to remoteusers as a VR environment. A VR visualization may comprise imagery of aVR environment. In the VR environment, the real-world objects seen by anMR participant may be represented by VR images of said real-worldobjects. In some examples, a VR presentation presented to the VR usermay comprise one of the MR views of one of the MR participants. However,in other examples, the VR presentation presented to the VR user maycomprise information from two or more MR views of two or more of the MRparticipants. With multiple MR views, the creation of an accurate VRrepresentation of the procedure may be enhanced relative to the use of asingle MR view to render the VR presentation. For example, MRparticipants may provide input for room mapping so that the room can bemapped and presented as a virtual environment to a VR participant.

FIG. 115 is a block diagram illustrating a distributed MR system 11500that includes one or more users at local environment 11508 that are incommunication with one or more users in a remote environment 11502. Thecommunication may occur over any type of network (e.g., shown as network11516). In some example, local environment 11508 may comprise anoperating room. MR devices 1-3 (11510, 11512 and 11514) may correspondto users in the operating room, such as a physician, nurse, medicaltechnician, anesthesiologist, or other users.

In general, local environment 11508 may be an environment where usersare in viewable proximity to a patient, whereas remote environment 11502may correspond to a location where users are not in viewable proximityto a patient. Since remote environment 11502 is not at the same locationas local environment 11508, any user of system 11500 at remoteenvironment 11502 may operate in a purely virtual reality. VR device11504 may facilitate this virtual reality environment, which may be avirtual reality world that is generated based on views and data from oneor more of the MR devices in local environment 11508 (e.g., roommapping).

As shown in FIG. 115, VR device 1 (11504) includes a view sharing window11506. According to this disclosure, VR device 1 (11504) is configuredto present one or more of the views of MR devices in the localenvironment 11508 in view sharing window 11506. Although shown as a viewsharing window (11404), the view sharing may be presented in other ways,such as via a remotely located MR device with a view sharing window orby VR device 11504 assuming the view of a selected one of the MR devicesin local environment 11508.

In some examples, a surgeon may perform one or more procedures on apatient in an operating room, with the aid of MR. During the procedure,the MR environment may be captured and presented to remote users viadisplay screens or as a VR environment. The surgeon may be able to viewthe patient directly in the operating room via an MR device, such aswith a visualization device 213. The MR users may utilize visualizationdevices that deliver mixed or augmented reality to the user, whereasremote VR users may utilize a visualization device that is pure virtualreality.

Alternatively, rather than using VR, remote users may use MR device thatare located remotely or may simply view display screens (such as desktopcomputers, laptop computers, or smartphones) that include camera feedsfrom cameras or the MR devices used by local participants. In someexamples, the remote display screens used by one or more remote usersmay show the same virtual elements that are shown to local MRparticipants. The remote participant may be able to select between MRviews and switch them at will, and the views selected by the remoteparticipant may include the virtual elements that are shown to the localMR participants that have that same corresponding local views that areselected by the remote participant.

In some examples, the MR systems can capture the images viewed by thesurgeon and then use those captured images to render VR images that canbe displayed to a remote spectator via VR. In this way, the remotespectator may be presented with a VR environment that is similar oridentical to the MR environment viewed by the local surgeon. Of course,the view of the VR participant may not be available to MR participants,since the VR participant is typically not present in the operating room.

When one or more MR participants and one or more VR participants areusing the system, the MR and VR participants may also be presented toone another in the MR and VR environments. For example, a surgeonworking in the operating room with the aid of MR may be presented withan MR environment that includes one or more VR participants, which maybe shown as MR objects (e.g., avatars) in the MR environment. Similarly,the VR participants may be presented with a VR environment that includesthe same objects and images shown in the MR environment. Thus, realobjects shown in the MR environment to MR users may be presented as VRobjects in the VR environment to VR users. In this way, the MR and VRworlds can be intertwined such that MR participants can see the VRparticipants as VR objects in the MR world. Similarly, VR participantscan view the MR objects and MR participants (e.g., avatars), as well asreal-world objects as VR objects in the VR world. When objects move inthe MR world, the object movements are captured by one or more of the MRparticipants. The VR world is then adjusted to reflect the changes inthe MR world that is captured by the MR participants. In this way, theVR participants are able to view a VR world, in real-time, that is basedon the MR world. Again, in some examples, the VR presentation to the VRuser may comprise one of the MR views of one of the MR participants.However, in other examples, the VR presentation to the VR user maycomprise information from two or more MR views of two or more of the MRparticipants. With multiple MR views, the creation of an accurate VRrepresentation of the procedure may be enhanced relative to the use of asingle MR view to render the VR. For example, multiple MR participantsmay provide input for room mapping so that the room can be mapped andpresented as a virtual environment to a VR participant.

As mentioned, one or more of the VR participants can be located remotelyrelative to the MR setting, such as an operating room or a remotelocation for patient interaction. This allows the VR participant to be aremote participant relative to the location of the patient encounter,procedure, or surgery. The ability to accommodate a VR participant canbe highly desirable for many situations and settings. For example,surgical experts may be consulted as VR participants in an otherwise MRprocedure in the operating room, thereby providing more expertise to theprocedure by virtue of the remotely-located VR participant. Forinstance, in this example, a user of an MR visualization device in theoperating room may request a remote surgical expert for a consultationduring the surgery. In this example, the remote surgical expert may usea VR visualization device to obtain a VR visualization of a scene in theoperating room. A local physician using MR could then ask questions orreceive guidance from a remote VR participant so as to help improve themedical process. A remote VR participant may be able to see what thelocal MR surgeon is seeing, e.g., by selecting the surgeon's MR view asthe VR view for the remote VR participant. In some examples, the remoteVR participant may be a surgical expert that is summoned during amedical procedure in order to gain insight or advice on the procedure.

In some examples, a MR system comprises a first MR device (e.g., avisualization device 213) configured to provide first medicalinformation about an orthopedic medical procedure to a first user via afirst MR presentation, and a second MR device (e.g., anothervisualization device 213) configured to provide second medicalinformation about the orthopedic medical procedure to a second user viaa second MR presentation. The first and second medical information, forexample, may comprise surgical guidance information in the form of oneor more virtual elements presented in the first or second MRpresentations. In some examples, the first medical information is thesame as the second medical information. In some examples, the firstmedical information and the second medical information are different.The virtual elements, for example, may comprise any of the virtualinformation described in this disclosure, such as virtual planes,virtual axes, virtual menus or widgets, or any virtual information thatmay be useful in the surgical procedure. The MR system may furtherinclude a VR device configured to present a VR presentation thatprovides at least some of the first or second medical information aboutthe orthopedic medical procedure to a third user. The VR presentationmay be based at least in part on the first MR presentation or the secondMR presentation. In some examples, the VR presentation comprises one ofthe first MR presentation and the second MR presentation, and in someexamples, the VR presentation is selectable on the VR device between thefirst MR presentation and the second MR presentation. The first user andthe second user may be located in viewable proximity to a patent and thethird user may be located remotely relative to the patient. The VRdevice may be configured to present the VR presentation to the thirduser in substantially real-time relative to MR presentation. The firstMR device may be configured to present the third user an avatar in thefirst MR presentation and the second MR device is configured to presentthe third user an avatar in the second MR presentation. In addition, theVR device may be configured to present the first and second users asavatars in the VR presentation.

In some examples, this disclosure describes a VR device for use by aremote medical professional, the VR device comprising a displayconfigured to present a VR presentation that provides medicalinformation associated with a patient to the remote medicalprofessional, wherein the VR presentation is based at least in part on aMR presentation, and one or more processors that control the display,wherein the MR presentation is captured locally by a local user of an MRdevice located within viewable proximity to the patient.

In some examples, an MR system comprises a first MR device configured topresent first medical information and first real-world information to afirst user via a first MR presentation, a second MR device configured toprovide second medical information and second real-world information toa second user via a second MR presentation; and a third deviceconfigured to provide third information to a third user, wherein thethird information is based at least in part on the first MR presentationor the second MR presentation. In some examples, the first medicalinformation is the same as the second medical information. In someexamples, the first medical information and the second medicalinformation are different. The third device may be configured to allowfor selection between the first MR presentation and the second MRpresentation on the third device. The first user and the second user maybe located in viewable proximity to a patient and the third user may belocated remotely relative to the patient. In some examples, the thirddevice presents the third information to the third user in real-timerelative to the first MR presentation on the first MR device and thesecond MR presentation on the second device. The third device maycomprise a display screen that presents the third information, oralternatively, the third device may comprise a VR device that presentsthe third information.

In some examples, a method according to this disclosure may comprisepresenting first medical information about an orthopedic medicalprocedure to a first user via a first mixed reality presentation on afirst visualization device, presenting second medical information aboutthe orthopedic medical procedure to a second user via a second MRpresentation on a second visualization device, and presenting a VRpresentation that provides at least some of the first or second medicalinformation about the orthopedic medical procedure to a third user of aVR device, wherein the VR presentation is based at least in part on thefirst MR presentation or the second MR presentation. In some examples,the first medical information is the same as the second medicalinformation. In some examples, the first medical information and thesecond medical information are different.

These and other methods described in this disclosure may be executed byone or more processors, in which case the techniques may be embodied inprocessor-executable instructions stored in a computer readable medium.The one or more processors may comprise one or more processors ofvisualization device 213 and the computer-readable storage medium maycomprise a memory of visualization device 213.

Many of the examples of view sharing and the participation of remoteparticipants to a procedure have described in this disclosure have beendescribed in the context of a surgical process. However, view sharingand the use of remote participants may also be used in other settings,such as in a pre-operative encounter with the patient, a post-operativeencounter, or other settings. For example, in a pre-operative orpost-operative encounter, a remote participant may be consulted (e.g.,with the aid of MR, VR or other techniques described herein) in order toleverage the expertise of that remote participant, who may be a medicalexpert, a medical device technician, a surgical expert, a nurse, or anyother person that may need to be consulted. In general, view sharing andthe participation of remote participants to a medical procedure may bedesirable in a wide variety of settings, including operating roomsettings, educational settings, pre-operative meetings or settings,post-operative meetings or settings, physical therapy settings, or othersettings.

As another example, MR may be used locally in the field for patientencounters, checkups, or field emergencies, such as for a remote checkupwith a patient at a patient's residence or for emergency care. Referringagain to FIG. 115, for example, local environment 11508 may comprise anylocation where patient care is needed. In these types of examples, aremote participant may be consulted in real time during this remotecheckup or emergency care with the use of VR device 11504. In suchexamples, the MR participant(s) may comprise a physician, nurse, orother person that interacts locally with the patient, e.g., at thepatient's home. The objects and images viewed by the MR participant maybe actual objects mixed with virtual objects. These same objects andimages may also be presented to a remote participant with the use of VRby presenting objects as virtual objects to the remote participant. Thiscan allow medical personal (working with AR) to leverage the opinionsand expertise of a remote VR participant in real time. In this way, a VRparticipant need not be present at the location of the patient in orderto view, diagnose, and help with patient care. In other examples, theremote participant may simply view one or more display screens, whichmay include information that is captured by MR participants or othercameras.

In some examples, the views and images captured by one or more MRparticipants may be viewed by a remote VR participant. For example, aphysician using VR may be able to access the views and images seenlocally by an MR participant to the procedure. Using VR, the physicianmay be presented with the views and images seen locally by the MRparticipant. In this way, the physician may be located remotely relativeto the patient but may view real-time objects and images of the patientand the patient's environment so as to aid and facilitate remotediagnosis and treatment by the physician using VR.

In some examples, the view of one or more MR users may be a selectablechoice for one or more remote users. In this way, multiple differentview options associated with multiple different MR perspectives may beaccessed by remote users. For example, a remote physician using VR maybe able to select and see the view of a first MR participant or a secondMR participant. In this way, the VR world viewed by the remote user maycomprise the same MR view shown by a local MR participant to theprocedure. In addition, the remote VR user may be able to change viewsto present a different MR view associated with a different MRparticipant to the procedure. Selectable views for the VR participantmay be desirable for operating room environments as well as remotepatient checkups and remote emergencies in the field.

In order to select the view of other users, users may select avatars,sub-windows, gaze lines, icons, or selections from drop down menus.These or other types of control mechanisms may be implemented as virtualobjects presented on a visualization device in a user's MR or VRpresentation. Then, when a first user selects the view of a second user,that second user's view may be presented to the first user. In this way,view sharing and the selectability of the view of other users mayprovide more information to MR users.

Relative to a local process or procedure, the VR participant may bepresented as an MR object (e.g., an avatar) in the MR world, which isshown to the MR participants. In this way, the VR participant may bepresented as being “in the room” relative to the MR participants. Thus,the VR participant can appear to be more integrated into the processwith the MR participant, giving the impression to MR participants thatthe VR participant is present, e.g., in the operating room or at thelocation of care. MR and VR participants may interact and even view atone another, as though they are interacting in the same room. The MRparticipant, however, is merely seeing an object (e.g., an avatar) ofthe VR participant and vice versa.

In some examples, a VR participant is able to interact or select fromviews associated with a plurality of MR participants. The VR participantmay be able to view a first VR world defined by the views of a first MRparticipant, and the VR participant may then be able to change views tothat captured by a second MR participant. In this way, for example, a VRparticipant may be given more control over the views seen in real time.Also, in yet additional examples, the room may be mapped and presentedto the VR participant in the VR world based on multiple MR views frommultiple local users. In this way, the VR world can be a more accuratereflection of the MR world and the real world seen by the multiple MRusers. Object rendition in the VR world may be determined by a votingscheme or another prioritization scheme such that objects seen bymultiple MR users are more likely to be shown in the VR world thanobjects seen by only one MR user.

In some examples, the VR participant may comprise a check on the workperformed by a local MR physician. In this example, the VR participantmay view and monitor a medical process performed by the local MRphysician. The VR participant may communicate and provide verbalfeedback to the local MR physician, while observing the views of thelocal MR physician in real time. This types of collaboration and safetychecks by VR participants relative to the work of local MR physicianscan be helpful for a positive patient outcome. Audio input may beprovided by MR or VR users and all users may hear the audio input asthough all users reside in the same room. This can provide more a moreintimate experience for the patient and MR users relative to thepresence of the VR user.

The level of control that is given to the remote VR participant may varyin different scenarios and settings. In some examples, the VRparticipant may be given a check on one or more steps of a procedure,such as a surgery. In this example, completion of a step of theprocedure may be conditioned on the VR participant providing approval ofcompletion of that step of the procedure. In this way, a local surgeonusing MR may be guided and aided by the expertise of a remote physicianusing VR. In some examples, the MR system may require input from theremote physician using VR such that that the remote physician needs toapprove the procedure or step performed locally by the local physicianusing MR. The remote physician using VR can watch the process in realtime, provide feedback, and decide whether the process or step has beenperformed properly by the local surgeon.

In still other examples, an MR system may comprise local MR users at thelocation of the patient, and a remote observer may be able to view theMR environment via one or more remote display screens. In this case, theremote observer may be able to select the views of the local MR users,or possibly select views associated with stationary cameras in thelocation of the patient. In this way, a remote participant may be ableto select different views and thereby gain an appreciation orunderstanding of the procedure. For example, the remote participant maycomprise a physician, who may be consulted during the procedure. Asanother example, the remote participant may comprise a medicaltechnician, such as a medical device technician, who may be consultedregarding information about the device or the implantation procedure. Ingeneral, “local” MR users may be users that are in viewable proximity toa patient or procedure, whereas “remote” users may be users that are notin viewable proximity to a patient or procedure. In some examples, thesystem can reduce the need for assistance from a medical devicetechnician, and possibly make the attendance of the medical devicetechnician unnecessary in some surgeries.

FIG. 116 is another block diagram illustrating an MR system 11600 thatincludes one or more users at local environment 11608 that are incommunication with one or more users in a remote environment 11602. Thecommunication may occur over any type of network (e.g., shown as network11616). In some examples, local environment 11508 may comprise anoperating room. MR devices 1-3 (11610, 11612 and 11614) may correspondto users in the operating room, such as a physician, nurse, medicaltechnician, anesthesiologist, or other users. In some examples, FIG. 116may be considered an instance of the scenario of FIG. 18, in whichmultiple users may view the same MR intraoperative guidance content1802.

As with the example of FIG. 115, in FIG. 116, local environment 11608may be an environment where users are in viewable proximity to apatient, whereas remote environment 11602 may correspond to a locationwhere users are not in viewable proximity to a patient. Since remoteenvironment 11602 is not at the same location as local environment11608, any user of system 11600 at remote environment 11502 may needaccess to views or data from the MR devices in local environment 11608.

As shown in FIG. 116, a user at remote environment 11602 may utilize adisplay 11604 configured to include a view sharing window 11606. Display11604 and view sharing window 11606 may be controlled or driven by oneor more processors (not shown). In any case, display 11604 is configuredto present one or more of the views of MR devices in the localenvironment 11608 in view sharing window 11606. Although shown as a viewsharing window (11404), the view sharing may be presented in other ways,such as display 11604 assuming the view of a selected one of the MRdevices in local environment 11608. FIG. 116 generally shows how aremote participant could use view sharing without the need for virtualreality. Instead, remote environment 11602, in this example, may simplyuse one or more display screens to convey information from localenvironment 11608 to one or more users at remote environment 11602.

In another example, an MR system that includes both MR users and remoteusers (e.g., VR users or remote users that view one or more displayscreens) may be used to facilitate remote patient checkups. In thiscase, a remote user may be leveraged to assist an MR user with a localpatient encounter via an MR system. For example, a nurse using MR mayinvolve a remote physician using VR or remote display screens, to aid ina patient encounter. The MR environment may include a nurse and patientusing MR, and a remotely located physician using VR (such as shown inFIG. 116) or display screens that display the MR views that are capturedlocally (such as shown in FIG. 116). In the case of the remote physicianusing VR (such as shown in FIG. 116), the patient may interact with theremote physician as though the physician was in the room with the nurseand patient. In such cases, the physician may leverage the MR views ofthe nurse in order to deliver medical advice to the patient.

In another yet example, an MR system that includes both MR and remoteusers (such as VR users or users that have display screens that displaythe MR views of others) may be used to facilitate emergency care. Inthis case, a remote user may be leveraged to assist an MR user withemergency care via an MR system. For example, an emergency medicaltechnician (EMT) using MR may involve a remote physician (using VR ordisplay screens that show local MR views) to aid in an emergency patientencounter. The physician may leverage the MR views of the EMT in orderto deliver immediate medical diagnosis for the patient. The MRenvironment may include the EMT using MR and a remotely locatedphysician using VR in order to give the perception that the remotephysician is “in the field” for the emergency care.

As mentioned elsewhere in this disclosure, an MR system can includemultiple visualization devices so that multiple users can simultaneouslysee the same images and share the same 3D scene, such as MRintraoperative guidance content 1802 (FIG. 18). In such examples, one ofthe visualization devices can be designated as the master device and theother visualization devices can be designated as observers. Any observerdevice can be re-designated as the master device at any time, as may bedesired by the users of the MR system.

With any of the examples described herein, it may also be desirable tohave one master device and many observer devices. Moreover, sinceassignment of the master device may change, the master role may bereassigned to different observers, essentially passing the master roleamong persons in the room. In one example of an orthopedic shoulderreconstruction surgery, one person may perform a drilling operation topinpoint a location on a patient's glenoid bone, a different person mayperform a reaming operation on the patient's glenoid bone, and then athird person may place an implant at the identified and reamed locationof the patient's glenoid bone. In this example or other surgicalexamples with multiple participants, each person may be assigned as amaster with regard to one or more steps of the medical procedure thatthey perform, and each person may be assigned as an observer with regardto the other steps, which are performed by others. In one example, themaster may be allowed to voluntarily accept from or relinquish themaster control over to the MR system. In another example, master controlmay be assigned or re-assigned automatically based on a specific masterassignment procedure, such as a procedure that requires approval of twoor more (or possibly a majority) of MR participants in order to change aspecific user to be the master.

In yet another example, a local surgeon could perform surgical cuts, andanother local or remote person (using VR) could perform one or moreregistration steps of the procedure, such as “SET,” “ADJUST,” and“MATCH” as described in this disclosure, e.g., for placement of MRobjects in a field of view for registration with physical anatomicalobjects. In other words, registration of a 3D model may be performedlocally by an expert as an MR participant or possibly even remotely byan expert as a VR participant. In these examples, the expert may be amedical device representative and the expert may be an MR or VRparticipant assigned to perform an initialization stage of a 3D modelwith real bone or tissue of the patient.

If the expert is a VR participant, the expert may view images from oneor more of the MR participants in order to view real time images of thepatient's bone structure relative to a 3D model that is superimposed onthe patient's bone structure. The expert participant may use commands,hand gestures, gaze or other control mechanisms to orient the 3D modelrelative to the patient's bone structure shown to the remote physicianas VR elements. The use of VR to accommodate a remote participant mayallow for a more qualified physician to perform the initializationstage. Alternatively, the expert may be a local MR participant, in whichcase it may still be advantageous to assign an initialization process ofa registration process to that expert. Then, after initialization, oneof the MR or VR users may initiate an optimization algorithm, such as aminimization algorithm to more precisely match the 3D model with realbone of the patient. The ability to involve a remote expert to asurgical procedure may be especially helpful for complex multi-stepsurgical procedures, such as a shoulder arthroplasty, an anklearthroplasty, or any other type of orthopedic surgery that requires oneor more complex steps.

Computer-aided steps, in particular, may be well-suited for a remotephysician, whereas physical steps (such as cutting, reaming, drilling orother steps) may need a physical presence by the physician.Initialization steps, for example, may be well-suited for remoteexecution by a VR participant or a local expert such as a medical devicetechnician. In some examples, one or all of the virtual objectregistration steps, e.g., of “SET,” “ADJUST,” and “MATCH” as describedin this disclosure, may be well-suited for remote execution by a VRparticipant or by a local participant that is an expert with the MRsystem or medical devices used in the procedure, such as a medicaldevice technician.

Furthermore, as mentioned elsewhere in this disclosure, the imagesdisplayed on a UI of an MR system (e.g., UI 522 of FIG. 5) can be viewedoutside or within the operating environment and, in spectator mode, canbe viewed by multiple users outside and within the operating environmentat the same time.

In some examples, an MR system, such as MR system 212, may support theability to view share amongst MR devices worn by different participantswithin the operating room. Moreover, view sharing may also beimplemented with one or more fixed cameras in the operating room, e.g.,allowing a physician to have immediate and re-occurring access to aspecific view of fixed cameras or other users. In this way, for example,a physician may be able to gain visual information very quickly, withoutneeding to move around or change his or her view.

For example, the physician (or other user) involved in an orthopedicsurgery may be presented with a mixed reality view from his or herperspective, but that user may be able to select and view the views ofother persons (or fixed cameras) within the operating room. This canallow the physician to quickly gain other perspectives within theoperating room. The views of other persons may comprise mixed realityviews that include real objects and superimposed virtual objects as partof that user's mixed reality presentation. When the physician (or otheruser) selects the view of another person in the physician's mixedreality presentation, e.g., on visualization device 213, that view ofthe other person may be presented to the physician on the physician's MRdevice, e.g., as a display window or an entire view on the MR device.

A shared view may include one or more real objects and one or morevirtual objects. The virtual objects shown in each user's MRvisualization device 213 may differ since different users have differentroles in the medical procedure, but when view sharing is implemented,the virtual objects of one user's view may be shared with other users.In this way, view sharing may allow for users to share unique MRpresentations that include virtual elements that might otherwise not beviewable to a user if view sharing was not available.

For example, a nurse and a physician may be presented with different MRpresentations, but when the physician selects the view of the nurse, thenurse's view may be presented to the physician and may includenurse-specific virtual elements that might otherwise not be presented inthe physician's MR presentation. Accordingly, by presenting the views ofother users, the physician may be able to gain different visualperspectives, different angles of view, and different virtual elementspresented in the view.

When different users are presented with different MR presentations, thevirtual elements presented to such users may differ. This allows thevirtual elements seen by one user to be specific to that user, andallows the virtual elements seen by another user to be specific for thatother user. Again, for example, physicians and nurses may be presentedwith different MR presentations that include virtual elements definedfor the doctor and for the nurse. In some examples, view sharing mayallow the physician to see the MR presentation of the nurse. Or in someexamples, view sharing may be implemented on an element-by-elementbasis. That is to say, one or more virtual elements presented to thenurse may be enabled or disabled by the physician so that the physicianis able to view selectively share with the nurse with respect tospecific virtual elements.

For example, tool identification elements may be presented to the nursein the nurse's MR presentation to aid the nurse with tool selection andtool tracking. Such tool identification elements (or other nursespecific elements) may be initially hidden from the physician. Withobject-specific view sharing, however, the physician may be able toenable or disable the virtual elements of the nurse on the physician'sMR presentation. Accordingly, the physician (or other user) can be ableto enable and disable virtual elements seen by other users so as tocustomize the MR presentation of that physician (or another MR user).

In some examples, an MR system may be used for preoperative,intraoperative or postoperative education purposes. The educationalbenefits may be provided to the patient, or in some cases, theeducational benefits may be provided to third-party observers, such asmedical students or other physicians. For example, an MR system may beused to provide education to a remote observer to the process. In thisexample, the remote observer may implement a VR device, such asdescribed herein. Alternatively, the remote observer may merely have theability to view MR feeds of local MR participants.

MR visualization devices may allow surgeons to view 3D content andvisualize exactly what will happen, step-by-step, during a procedure.The step-by-step explanation may be virtual content provided to the MRvisualization device used by the physician. Information provided to thephysician may comprise general patient information based on informationfrom the general patient population or may be patient-specificinformation associated with the specific patient on which a givenprocedure will be performed.

In some examples, an MR visualization device such as visualizationdevice 213 can be configured to present a virtual person to the MRvisualization device used by the physician, the patient, or anotheruser, and the virtual person may explain one or more details about thesurgery or other medical procedure to be performed. The virtual personmay be an automated recording, or may comprise an actual physician,located remotely and operating with a remote VR device that interactswith MR visualization devices in the operating room. Videos orinstructional queues may be selectable widgets to the user of MRvisualization device, and such videos or instructional queues may bedefined for different steps or stages of a medical procedure. This typeof interactive and selectable real-time intraoperative instruction maybe especially useful to aid surgeons that do not perform the procedureregularly, frequently, or on a higher volume basis.

In some examples, an MR visualization device (e.g., visualization device213) may allow the physician to place virtual bone, virtual implants orother virtual elements on the patient for educational or planningpurposes. For example, actual patient images or segmentations of thepatient's bones or other physical features may be presented in AR, andvirtual implants, jigs or other elements may be placed on the patientvirtually by the physician. The virtual placement of implants or jigsrelative to the patient may improve the sizing and selection of theimplants or jigs or may facilitate or improve patient-specific implantsor jigs that accommodate unique anatomical features of the patient.

The physician may view virtual 3D images of implants, jigs or otherelements from different perspectives, and may manipulate the 3D imagesrelative to images or segmentations of the patient using hand gestures,voice commands, gaze direction and/or other control inputs. For example,the physician may place a virtual representation of an implant or jiginto an implantation location of the patient, e.g., in the patient'sshoulder. Placement may involve setting and adjusting the virtualimplant via “SET” and “ADJUST” techniques, e.g., as described elsewherein this disclosure in relation to the example of MR-based intraoperativeguidance for glenoid bone procedures. Once placed in an initializationprocess, a matching algorithm may be implemented to “MATCH” theimplantation to the patient's anatomy via a computer algorithm, e.g., asdescribed elsewhere in this disclosure in relation to the example ofMR-based intraoperative guidance for glenoid bone procedures.

Moreover, as described elsewhere in this disclosure, virtualthree-dimensional representations may be defined in the MR presentationto aid the physician with the procedure. The virtual 3D representationsmay illustrate a cutting axis relative to an anatomical element, such asa cutting axis on a humoral bone for shoulder reconstruction surgery. Asanother example, the virtual 3D representations may illustrate a reamingaccess relative to an anatomical element, such as a reaming axisrelative to a patient's glenoid bone for shoulder reconstructionsurgery. As yet another example, the three-dimensional representationsmay illustrate a drilling axis relative to an anatomical element. Ofcourse, the virtual 3D representations may be defined by the surgicalprocedure being performed. In ankle surgery, for example, virtual 3Drepresentations may include virtual elements presented relative to avirtual ankle model or a model of the talus and/or tibia.

As still another example, the three-dimensional representations mayillustrate placement of a virtual jig relative to the anatomicalelement, and in this case, the three-dimensional representations mayalso illustrate an axis (e.g., a reaming axis) relative to the virtualjig. In these examples, the anatomical element may be a real bone of thepatient, or it may comprise a 3D virtual representation of an anatomicalfeature of a patient that is generated based on one or more real imagesof the anatomical feature of the patient. The 3D virtual representationof an anatomical feature may be registered to the actual anatomicalelement of the patient, i.e., the real anatomy of the patient, so thatit is shown as an overlay in an MR view of the actual, physicalanatomical element. In this way, preoperative planning associated withthe 3D virtual representation (e.g., cutting axes, reaming axes,drilling locations, virtual jigs, or other features) may be defined withregard to the 3D virtual representation so that when the 3D virtualrepresentation is registered to the real anatomical feature of thepatient, the preoperative planning features (e.g., cutting axes, reamingaxes, drilling locations, virtual jigs, or other features) are alsoaligned properly with regard to the real anatomical feature of thepatient.

In some examples, an MR system may allow for educational collaborationwith a patient or with other medical professionals. The MR-basedcollaboration may occur during a preoperative meeting with the patientin order to explain the procedure to the patient with the aid of MR. TheMR-based collaboration may use real patient imagery in combination withvirtual representations of implants, jigs or other medical devices, inorder to illustrate the details of the procedure to the patient.

In other examples, MR system may allow for postoperative case analysis.In this case, actual images of surgical results may be compared withimages of other surgical results of other patients. As another example,actual images of surgical results may be compared with virtual images inorder to provide for postoperative assessment. Postoperativecollaboration, like the preoperative collaboration may be performed withthe patient or with other medical professionals all participating inmixed reality with MR visualization devices or VR visualization deviceslike those described herein.

In still other examples, MR systems may allow for educational trainingduring a medical procedure or surgery. That is to say, one or morepassive viewers to the procedure may have passive MR visualizationdevices that provide the user with a view of the MR environment withoutany control over the MR environment. Such passive viewers may be locatedin the operating room, or in a viewing room adjacent the operating room,or remotely from the operating room. Alternatively, in some cases, apassive viewer may be located remotely, essentially viewing one or moreof the MR presentations seen by persons in the operating room. Suchpassive viewing from a remote location without any presence orcollaboration with members in the operating room may be used fortraining and educational purposes. Such an MR viewing tool may be veryuseful for use in either real time or delayed viewing of the procedure.

In some examples, an MR system may maintain a database of informationassociated with prior patients or prior surgeries, e.g., stored in astorage system of visualization device 213 or stored in a database(e.g., storage system 206 of FIG. 2) communicatively coupled tovisualization device 213. In this case, the database of information maybe used to identify situations or circumstances of a prior surgery orprocedure that may impart knowledge or information germane to a currentsurgery or procedure. Particular parameters, such as patientmeasurements, anatomical feature sizes, shapes or anomalies, sizes ofimplants or jigs, or other parameters may be used for identifyingsimilarities or matches between prior surgeries or procedures stored inthe database and the current procedure.

In this way, an MR system can be used to help a physician identify casesimilarities between a current procedure associated with a currentpatient and past procedures performed on past patients. In some cases,an MR system may be configured to present surgical recommendations tothe user physician (such as sizes and shapes of implants, types ofimplants, recommendations for anatomical or reverse anatomicalprocedures, or other types of recommendations). The recommendations, forexample, may be defined by machine learning techniques that leverage thedata stored in the database.

During intraoperative phase 306 (FIG. 3), a surgeon may perform anorthopedic surgery that follows a complex series of workflow steps, someor all of which may proceed according to prescribed sequence. Althoughsurgeons are typically well trained to perform the steps of anorthopedic surgery, incidents may occur where one or more steps of theorthopedic surgery are accidentally omitted or performed in an incorrectmanner. Moreover, incidents may occur where the surgeon remembers toperform a step of an orthopedic surgery, but misremembers how to performone or more aspects the step, or would otherwise benefit from a reminderof how to perform the step or guidance about one or more aspects of thestep, some of which may vary according to a specific surgical plangenerated for a particular patient.

Such incidents could impact surgical procedure efficiency, surgicalprocedure efficacy or patient outcomes. Paper checklists may be used toremind surgeons and operating room (OR) staff to perform the steps ofthe surgery. However, there are numerous drawbacks to the use of paperchecklists. For example, it might not be easy for a surgeon to see apaper checklist, especially when the surgeon is holding surgical tools.Moreover, because the surgeon must preserve sterility, it may beimpossible for the surgeon to use her or his hands to mark off items inthe paper checklist. Additionally, a paper checklist includes visualclutter regarding completed and upcoming steps and it may take thesurgeon valuable time to process the paper checklist. Moreover, thevisual clutter in the paper checklist may allow the surgeon to visuallyskip over a step of the surgery. Having a nurse maintain and read backthe checklist to the surgeon may increase expenses because this approachmay require a dedicated nurse to be present for this purpose.Furthermore, a paper checklist does not allow the surgeon to access anymore information than the information that is provided on the paperitself.

Presenting a computerized version of a paper checklist on a computerscreen in the OR may have many of the same problems as a conventionalpaper checklist. For example, presenting all steps of the surgery on thecomputer screen may result in visual clutter that may delay or confusethe surgeon, and may require the surgeon to look away from a surgicalsite. Moreover, a dedicated nurse may be required to control thechecklist presented on the computer screen. Furthermore, although acomputerized version of a paper checklist may, in principle, allow thesurgeon to access more information during the surgery, the surgeon wouldhave difficulty requesting the display of such information. Forinstance, because of the need to preserve sterility, the surgeon may beunable to use a mouse or keyboard to control the computer. The surgeonmay be able to provide voice instructions to the computer or a nurse,but voice commands may be misinterpreted by computers and nurses, whichmay result in lost time, or may again require a dedicated nurse beavailable to control the computer just in case the surgeon wants thecomputer to display additional information.

An additional problem with checklists presented on computer screens isthat the computer screens are frequently somewhat distant from theorthopedic surgeon during a surgery in order to ensure that theorthopedic surgeon and other healthcare professionals have adequate roomto work around the patient. For instance, the computer monitors may bemounted to the walls of the operating room. However, this alone maycause certain problems for orthopedic surgeons. For instance, in oneexample, some orthopedic surgeons may wear magnifying loupe eyewear orstandard corrective eyewear to help the surgeon better see the surgicalsite on the patient. However, since the surgical site on the patient istypically much closer to the orthopedic surgeon than the location wherethe computer monitor is located, the surgeon's eyewear may make itdifficult to see the computer monitor that is at a further distance. Inother words, the difference in focal distance between the surgical siteand the computer monitor may be so great that the orthopedic surgeoncannot clearly see both the surgical site and the computer monitor whenwearing (or without wearing) particular eyewear. Thus, the surgeon mayneed to remove or change the eyewear in order to see the computermonitor. Removing or changing eyewear may consume time and may introduceanother potential vector for the spread of germs and other types ofcontamination.

This disclosure describes techniques that use extended reality (XR) toassist users (e.g., surgeons or other types of persons) through theworkflow steps of orthopedic surgeries in a way that may addresschallenges such as those mentioned above. As described elsewhere in thisdisclosure, XR may include VR, MR, and AR. In examples where XR is usedto assist a user through the workflow steps of an orthopedic surgery andXR takes the form of VR, the user may be performing a simulation of theorthopedic surgery or may be performing the orthopedic surgery remotely.In examples where XR is used to assist a user through workflow steps ofan orthopedic surgery and XR takes the form of MR or AR, the surgeon mayconcurrently perceive real-world objects and virtual objects during theorthopedic surgery.

For instance, FIG. 117 is a block diagram illustrating an example system11700 that may assist a user, such as a surgeon, nurse or other medicaltechnician, through steps in the workflow steps of orthopedic surgeries,in accordance with a technique of this disclosure. In the example ofFIG. 117, system 11700 includes an XR visualization device 11702, acommunication network 11704, and one or more computing systems 11706Athrough 11706N (collectively, “computing systems 11706”). In otherexamples, system 11700 may include more, fewer, or different devices andsystems.

Computing systems 11706 may include various types of computing devices,such as server computers, personal computers, smartphones, laptopcomputers, and other types of computing devices. For ease ofexplanation, this disclosure may describe actions performed byprocessing circuits, data storage systems, and communication interfacesof XR visualization device 11702 and computing systems 11706 as beingperformed by XR visualization device 11702 and computing systems 11706as a whole. Communication network 11704 may include various types ofnetworks, such as a local area network, the Internet, and/or other typesof communication networks. Various computing systems of orthopedicsurgical system 100 (FIG. 1) may include system 11700. For example,intraoperative guidance system 108 may include system 11700. In someexamples, XR visualization device 11702 may be implemented as shown inthe example of FIG. 5.

In the example of FIG. 117, XR visualization device 11702 may be worn byan orthopedic surgeon and may output an XR visualization for display. Insome examples, XR visualization device 11702 is worn by another type ofuser (e.g., a nurse, medical technician, or other type of user) and mayoutput the XR visualization for display. XR visualization device 11702may access information about workflow steps of an orthopedic surgeryfrom one or more of computing systems 11706. XR visualization devicessubstantially similar or identical to XR visualization device 11702 maybe worn by other persons in a surgical operating room, such as one ormore nurses, one or more medical technicians, or one or more additionalsurgeons, such that one or multiple persons may observe AR, MR and/or VRimagery providing guidance regarding workflow steps of an orthopedicsurgery.

In an example, the XR visualization includes a set of one or morevirtual checklist items. Each of the one or more virtual checklist itemscorresponds to an item in a checklist of steps of an orthopedic surgery(e.g., a shoulder arthroplasty, an ankle arthroplasty, or any other typeof orthopedic surgery). The steps may be arranged in sequence accordingto an order in which the steps are to be performed or may be arrangedtopically into step topics according to physical or functionalcharacteristics of the procedural steps. Furthermore, in this example, acamera or other sensor of XR visualization device 11702 may detect acommand of the orthopedic surgeon or other user to select a virtualchecklist item in the set of virtual checklist items. In some examples,one or more devices separate from XR visualization device 11702 maydetect the command. For instance, one or more cameras, microphones, orother types of devices positioned in an operating room may detect thecommand. Nevertheless, for ease of explanation, this disclosuredescribes XR visualization device 11702 as detecting the command.However, discussion of XR visualization device 11702 detecting thecommand may apply to one or more other types of devices detecting thecommand. In response to detecting the contactless command, XRvisualization device 11702 may update the XR visualization to includeadditional information regarding a step of the orthopedic surgerycorresponding to the selected virtual checklist item.

In some examples, the command is a contactless command. The contactlesscommand is performed without the orthopedic surgeon or other usertouching any solid object. For example, the contactless command may be ahand gesture performed in the air. For instance, the contactless commandmay be a pinching gesture directed to a virtual element of the XRvisualization. In some examples, the contactless command is a voicecommand.

In other examples, XR visualization device 11702 may detect a commandthat involves contact. For example, XR visualization device 11702 maydetect a first command in the form of a hand gesture in which a user(e.g., the orthopedic surgeon or other type of user) taps one or morefingers of the user's left hand onto the back of the user's right hand.In this example, XR visualization device 11702 may detect a secondcommend in the form of a hand gesture in which the user taps one or morefingers of the user's right hand onto the back of the user's left hand.In another example, the user may tap on a surface (e.g., a sterilizedsurface), such as a corner of the operating table or a surgical item.

In examples where the command is a contactless command, because theorthopedic surgeon or other user is able to view the additionalinformation without touching any solid object, there is no additionalrisk of contamination from the orthopedic surgeon or other user touchinga physical computer user interface device, such as a mouse, keyboard ortouchscreen. Similar considerations regarding reduction of risk ofcontamination may apply with contact-based commands on surfaces that arealready sterilized, such as the surgeon's own hands or a sterilizedsurgical item. Additionally, in examples where the command (e.g., acontactless command) is a hand gesture, the use of the hand gesture toaccess the information may avoid the problems with voice commandsdiscussed above, although voice commands also may be used in someexamples. In this way, a computing system (e.g., XR visualization device11702, one of computing system 11706, etc.) that handles the workflowchecklist for the orthopedic surgery may be improved by decreasing thecomputing system's risk of acting as a vector for spreadingcontamination. Furthermore, because the virtual checklist items may bepresented in an XR visualization in a focal plane that is relativelyclose to a focal plane of the surgical site, the orthopedic surgeon maynot need to change or remove eyewear in order to see the virtualchecklist items. Thus, in various examples, the techniques of thisdisclosure may improve the accessibility of the computing system,especially for orthopedic surgeons or other users with visionimpairments and may enable the orthopedic surgeon or other users to skipsteps otherwise needed to access the virtual checklist items, therebyimproving efficiency of use of the computing system.

XR visualization device 11702 may present various virtual checklistitems in the XR visualization. For example, the XR visualization mayinclude virtual checklist items corresponding to a current step of theorthopedic surgery that the orthopedic surgeon is performing. In thisexample, the XR visualization may exclude virtual checklist itemscorresponding to completed steps of the orthopedic surgery. Furthermore,in some examples, the XR visualization may exclude virtual checklistitems corresponding to one or more steps of the orthopedic surgeryoccurring after the current step of the orthopedic surgery. Forinstance, in one example, XR visualization device 11702 may excludevirtual checklist items corresponding to all steps of the orthopedicsurgery occurring after the current step. In another example, the XRvisualization may exclude checklist items corresponding to all steps ofthe orthopedic surgery except a next step of the orthopedic surgery. Insome examples, the displayed set of virtual checklist items in the XRvisualization is user configurable.

There are different steps for different orthopedic surgeries. Forinstance, the series of steps to perform a knee arthroplasty isdifferent from the series of steps required to perform a reverseshoulder arthroplasty. In one example, the series of steps for ashoulder arthroplasty may include: an incision step, a step of severingthe humeral head, a step of drilling a hole for a reaming guide pin, astep for performing a glenoid reaming process, a step for installing aglenoid implant, a step for preparing the humerus, a step for installinga humeral implant, and a step for closing the incision. In someexamples, the steps may include nested sub-steps. In some examples, theseries of steps may include the graft, humerus cut, install guide,glenoid reaming, and glenoid implant steps of FIG. 10. In some suchexamples, the next step of the orthopedic surgery may be a sub-step.Steps such as those described above may be presented as checklist items,e.g., using AR, MR, or VR visualization. Of course, the steps for anklearthroplasty would also differ from those of other types of orthopedicsurgeries, and virtual workflow guidance and virtual checklists maydefined to guide and track the specific steps defined for a given typeof ankle arthroplasty or other orthopedic surgical procedure.

Moreover, an orthopedic surgery may be customized to a particularpatient. For example, an additional step of removing osteophytes may berequired in a shoulder arthroplasty surgery for one patient but mightnot be required in the shoulder arthroplasty surgery of another patient.That fact that there may be variations within the same surgery fordifferent patients may increase the mental load on surgeons andoperating room personnel. Accordingly, the XR visualization may includevirtual checklist items corresponding to patient-specific steps orsub-steps of the orthopedic surgery customized for the particularpatient. Including virtual checklist items corresponding topatient-specific steps or sub-steps may be helpful in reminding surgicalpersonnel of how the orthopedic surgery is to be performed for theindividual patient.

The XR visualization may present various types of information in thevirtual checklist items. For instance, in one example, a virtualchecklist item in the XR visualization may include text identifying acorresponding step of the orthopedic surgery. In some examples, avirtual checklist item in the XR visualization may include an icon orother non-text graphic representing the corresponding step of theorthopedic surgery. In some examples, a virtual checklist item mayspecify which surgical items (e.g., tools, implants, etc.) to use in thecorresponding step of the orthopedic surgery. The XR visualization maypresent the virtual checklist items according to various formats. Forexample, the XR visualization may present the virtual checklist items asa row or a column of items, each of which may contain text, icons, orother information. In some examples, the XR visualization may presentthe virtual checklist items in a 3-dimensional stack-of-cards format,where virtual checklist items are in different virtual cards that may beremoved or added to the stack.

In some examples, the virtual checklist items may correspond to steps ofthe orthopedic surgery that are specific to an individual patient. Inother words, a virtual checklist item may correspond to a step includedin a version of an orthopedic surgery specific to a patient that is notincluded in a version of the orthopedic surgery of other patients. Forexample, in the example of the shoulder arthroplasty surgery providedabove, the XR visualization may include a virtual checklist item thatcorresponds to a step of removing osteophytes during the surgery for afirst patient but does not include this virtual checklist item in thesurgery for a second patient who does not have osteophytes that need tobe removed.

XR visualization device 11702 may present various types of additionalinformation in the XR visualization. In one example, the additionalinformation may include additional text describing a step of theorthopedic surgery. For instance, the additional information may includepreviously prepared notes by the orthopedic surgeon or other person. Insome examples, the previously prepared notes may be specific to thepatient on which the orthopedic surgeon is operating. For instance, thepreviously prepared notes specific to the patient may tell theorthopedic surgeon to install screws in a particular way so as to avoidcysts present at particular positions in a bone of a first patient. Inthis example, other patients may not have cysts at the same positions asthe first patient. In some examples, the previously prepared notes maybe common across patients but specific to an individual surgeon,hospital, care network, or other grouping. For instance, a surgeon mayalways want to have access to his or her personal notes regarding aparticular step of the surgery.

In some examples, the additional information may include one or morevirtual 3-dimensional objects associated with the step of the orthopedicsurgery. In other words, XR visualization device 11702 may update the XRvisualization to include one or more virtual 3-dimensional objectsassociated with the step of the orthopedic surgery. The 3-dimensionalobjects may include one or more virtual 3-dimensional models of bonesinvolved with the step of the orthopedic surgery. For instance, in thecontext of a shoulder arthroplasty surgery described elsewhere in thisdisclosure, the 3-dimensional objects may include a 3-dimensional modelof a patient's scapula. In this example, the XR visualization mayinclude one or more virtual 3-dimensional objects that do not correspondto real-world objects. For instance, the virtual 3-dimensional objectsmay include a virtual reaming axis. The virtual 3-dimensional modelsshown in the XR visualization during the surgery may be the same asvirtual 3-dimensional models shown during preoperative phase 302. Thus,an orthopedic surgeon or other user may already be familiar with thevirtual 3-dimensional models because the orthopedic surgeon may haveused the virtual 3-dimensional models during the preoperative planningphase. Thus, in some examples, the XR visualization may include virtual3-dimensional models such as those shown in FIG. 15A, FIG. 15B, FIG.15C, FIG. 15D, FIG. 16, FIG. 17, FIG. 27, FIG. 28, FIG. 29, FIG. 30, andso on. In some examples, one or more of the 3-dimensional models isspecific to an individual patient undergoing surgery. In some examples,one or more of the 3-dimensional models is generic across patients.

In some examples, the XR visualization may also include virtual controlsfor controlling the virtual 3-dimensional objects. The orthopedicsurgeon or other user may use contactless commands (e.g., hand gesturesor voice commands) or contact-based commands to select the virtualcontrols. The virtual controls may allow the orthopedic surgeon or otheruser to hide or display individual virtual 3-dimensional objects, rotatethe virtual 3-dimensional objects, scale the virtual 3-dimensionalobjects or otherwise control the virtual 3-dimensional objects. FIG. 13,described in detail elsewhere in this disclosure, is an example of aninterface in a XR visualization that includes virtual controls forcontrolling the virtual 3-dimensional objects associated with a step ofan orthopedic surgery. Particularly, FIG. 13 is an example of aninterface in a XR visualization that includes virtual controls forcontrolling the virtual 3-dimensional objects associated with a glenoidreaming step of a shoulder arthroplasty surgery. In some examples, XRvisualization device 11702 may update the XR visualization to rotate orscale the one or more virtual 3-dimensional objects in response to acommand from the orthopedic surgeon or other user. In some suchexamples, the command may be a contactless command, such as a handgesture or a voice command.

Furthermore, in some examples, XR visualization device 11702 may updatethe XR visualization to show to the orthopedic surgeon or other user ananimation of the step of the orthopedic surgery as applied to the one ormore virtual 3-dimensional objects. The animation may show progressivemovement of one or more virtual 3D objects, such as tools or implants,from initial positions, to intermediate positions, and to finalpositions to represent the operations needed to complete the step in theprocedure. For example, in the context of a shoulder arthroplasty, theXR visualization may show an animation of how to perform a step ofcutting a humeral head. This may help the orthopedic surgeon or otheruser remember how to perform a step of the orthopedic surgery, or how toperform the step in a particular way in accordance with apatient-specific surgical plan. In some examples, the animation isspecific to an individual patient. In some examples, the animation isgeneric across a set of patients.

In some examples, XR visualization device 11702 may detect a command(e.g., a contactless command or contact-based command) from theorthopedic surgeon or other user to mark the step of the orthopedicsurgery complete. In examples where the command is a contactlesscommand, the contactless command may be a hand gesture or a voicecommand. Marking a step as complete may comprise storing data (e.g., byone of computing systems 11706) indicating that the step is complete.Furthermore, based on the command, XR visualization device 11702 mayupdate the XR visualization to include a virtual checklist itemcorresponding to a next step in the checklist of steps of the orthopedicsurgery. For instance, XR visualization device 11702 may update the XRvisualization to mark the current step of the orthopedic surgerycomplete in response to detecting the command from the orthopedicsurgeon.

In some examples, one or more of computing systems 11706 may displayvirtual checklist items to one or more persons other than the orthopedicsurgeon. For instance, an XR visualization device or computer maydisplay the same virtual checklist items that XR visualization device11702 displays to the orthopedic surgeon. This may allow the otherperson to track the progress of the surgery. Such other persons mayinclude nurses, product representatives, students, other surgeons, andso on. Techniques for intraoperative collaboration and education aredescribed elsewhere in this disclosure.

In some examples, a person other than the surgeon may be involved inmarking a step of the orthopedic surgery complete. For example, inresponse to detecting a command from the orthopedic surgeon to mark astep of the orthopedic surgery complete, a computing device (e.g., oneof computing systems 11706) associated with a second person may promptthe second person to confirm that the step of the orthopedic surgery iscomplete. In this example, the computing device associated with thesecond person is communicatively coupled to the XR visualization device(e.g., an MR or VR visualization device). Thus, operating room staff(e.g., surgeons, nurses, etc.) may communicate with and/or consult withremote users who may be able to view virtual checklist items.

For instance, the computing device associated with the second person maybe a XR visualization device worn by the second person, a smartphone ofthe second person, a computer used by the second person and so on. Thecomputing device associated with the second person may becommunicatively coupled to the XR visualization device via communicationnetwork 11704. In some examples, one or more other computing devices(e.g., other ones of computing systems 11706) may process informationfrom XR visualization device 11702 or other devices to generateinformation sent to the computing device associated with the secondperson. Receiving such information may cause the computing deviceassociated with the second person to generate the prompt.

In this example, as part of updating the XR visualization to include thevirtual checklist item corresponding to the next step in the checklistof steps of the orthopedic surgery, XR visualization device 11702 mayupdate the XR visualization to include the virtual checklist itemcorresponding to the next step in the checklist of steps of theorthopedic surgery in response to the second person confirming that thestep of the orthopedic surgery is actually complete. In this way, thesecond person has the opportunity to verify that the step of theorthopedic surgery is actually complete. This may help prevent theaccidental skipping of steps of the surgical procedure and/or ensurethat steps are performed according to general standards of care orpatient-specific requirements. In other examples, a person other thanthe orthopedic surgeon may make the initial command to mark a step ofthe orthopedic surgery complete and the surgeon or other person mayconfirm that the step is actually complete.

In some examples, to enable the second person to verify that a step iscomplete, the computing device associated with the second person maydisplay a video feed from a camera that is positioned to see thesurgical site. For instance, in one example, the camera is integratedinto XR visualization device 11702 worn by the orthopedic surgeon. Inanother example, the camera is mounted to a frame or wall of theoperating room.

In some examples, one of computing systems 11706 may control varioussurgical tools based on which step of the surgical procedure is thecurrent step of the surgical procedure. For example, the computingsystem may disable a particular tool if the tool is not supposed to beused during the current step. For instance, in the context of a shoulderarthroplasty surgery, the computing system may disable a reaming drillduring a step of cutting the patient's humeral head if the reaming drillshould not be used during the step of cutting the patient's humeralhead. This may help prevent the surgeon from skipping a step of theorthopedic surgery. To control which surgical tools may be used with aparticular step, computing system 11706 may access a predefined list ofsurgical tools that may be used in particular steps of the surgicalprocedure.

In some examples, the workflow management process described herein mayguide a process for helping a nurse provide the correct tools to thesurgeon, as described elsewhere in this disclosure. For instance, avirtual object in an XR visualization presented to the nurse mayindicate to the nurse which tools to provide to the surgeon for the nextstep of the orthopedic surgery. In another example, lights built intothe surgical tools or trays may indicate to the nurse which tools are tobe used in the next step of the orthopedic surgery. Such examples aredescribed in greater detail elsewhere in this disclosure.

In some examples, a computing system (e.g., one of computing systems11706) outputs checklist items for display to a nurse or otherindividual. In some such examples, rather than XR visualization device11702 detecting a command to mark a step of the orthopedic surgerycomplete, the computing system may mark the step of the orthopedicsurgery complete when the nurse uses a device of the computing system tooptically scan a code (e.g., a bar code) of a surgical item (e.g.,surgical tool, implant, box of surgical items, etc.) used in the nextstep of the orthopedic surgery. This may have the combined effect ofadvancing the checklist and updating an inventory of available surgicalitems. In this example, the computing system may warn the nurse if thenurse did not scan the expected surgical item for a next step of theorthopedic surgery.

In some examples, a computing system in system 11700 may record surgicalnotes and automatically associate the surgical notes with steps of theorthopedic surgery. For instance, the surgeon may dictate notes duringthe course of the orthopedic surgery. In this example, a microphone ofXR visualization device 11702 or one of the computing systems 11706 maydetect and record the surgeon's voice. One of computing systems 11706may store (and in some examples transcribe) the spoken notes. In someexamples, a nurse may transcribe the notes dictated by the surgeon.Additionally, the computing system may associate the notes dictatedwhile a particular step of the orthopedic surgery is the current stepwith the particular step. In other words, notes dictated during thesurgery may be associated with the steps of the surgery in which thenotes were dictated. Thus, the notes for particular steps of theorthopedic surgery may be retrieved based on the steps of the orthopedicsurgery.

Similarly, in some examples, one or more computing systems 11706 mayautomatically associate portions of a video of the surgery with thecorresponding step of the surgery. This may allow users to quickly jumpto portions of the video corresponding to steps of the surgery that areof interest to the users.

FIG. 118 is a flowchart illustrating an example operation to assistusers, such as surgeons, nurses, or other medical technicians, throughsteps in the workflows of orthopedic surgeries, in accordance with atechnique of this disclosure. In the example of FIG. 118, a XRvisualization device 11702 worn by an orthopedic surgeon may output a XRvisualization for display (11800). The XR visualization includes a setof one or more virtual checklist items. Each of the one or more virtualchecklist items corresponds to an item in a checklist of steps of anorthopedic surgery, such as ankle arthroplasty or shoulder arthroplasty.

Furthermore, XR visualization device 11702 may detect a contactlesscommand (e.g., a hand gesture or voice command) of the orthopedicsurgeon to select a virtual checklist item in the set of virtualchecklist items (11802). The contactless command is performed withoutthe orthopedic surgeon touching any solid object and may be detected bya camera, microphone, or other sensor associated with XR visualizationdevice 11702 worn by the surgeon, an XR visualization device worn byother person in the operating room, or another sensor not carried by anXR visualization device. Furthermore, in response to detecting thecontactless command, XR visualization device 11702 may update the XRvisualization to include additional information regarding a step of theorthopedic surgery corresponding to the selected virtual checklist item(11804).

In some examples, XR visualization device 11702 is a MR visualizationdevice or AR visualization device that the orthopedic surgeon wears orotherwise uses during surgery. In some examples, XR visualization device11702 is a VR visualization device. In such examples, the orthopedicsurgeon may use the VR visualization device to perform a simulation ofthe orthopedic surgery. During such a simulation, it may be helpful tothe orthopedic surgeon to see the virtual checklist items in the sameway that the orthopedic surgeon would see the virtual checklist items ina MR visualization device or an AR visualization device.

FIG. 119 is a conceptual diagram illustrating an example XRvisualization 11900 that includes a set of one or more virtual checklistitems, as viewed by a user, such as an orthopedic surgeon, nurse, orother medical technician, while performing an orthopedic surgery on ashoulder of a patient. In the example of FIG. 119, the patient is lyingon an operating table and the orthopedic surgeon is viewing the patientfrom above. The orthopedic surgeon has already made an incision 11902 inthe patient's shoulder and exposed a glenoid surface 11904.

In accordance with a technique of this disclosure, XR visualization11900 includes a set of virtual checklist items 11906. Virtual checklistitems 11906 correspond to items in a checklist of steps of theorthopedic surgery that the orthopedic surgeon is performing on thepatient. In the example of FIG. 119, virtual checklist items 11906 arepresented in the form of a stack of virtual cards, each corresponding toa step (or sub-step) of the orthopedic surgery. In other examples, oneor more virtual checklist items 11906 may be presented in an XRvisualization in a column form. In some examples, one or more virtualchecklist items 11906 are in a row form, similar to the arrangement ofselectable buttons 1002 in FIG. 10.

A virtual checklist item 11908 includes text describing a step of theorthopedic surgery and a graphic corresponding to the step. Inparticular, virtual checklist item 11908 includes text describing a stepfor installing a reaming guide pin. Furthermore, as shown in the exampleof FIG. 119, virtual checklist item 11908 includes patient-specificcontent, namely that the patient has a bone cyst on a coracoid processand to avoid accidental contact. Such patient-specific content would notbe present for all patients. In some examples, virtual checklist item11908 may include one or more warnings associated with the current step.For instance, a virtual checklist item associated with a step ofinstalling fixation screws 16102A and 16102B of FIG. 161 may include atextual or graphical warning against over torqueing fixation screws16102A and 16102B.

As described elsewhere in this disclosure, a surgical plan may bedeveloped for an orthopedic surgery during preoperative phase 302 (FIG.3). For instance, the surgical plan may be developed using theBLUEPRINT™ system. However, a surgeon may deviate from the surgical planwhen performing the orthopedic surgery. In other words, the surgeon'sactions during intraoperative phase 306 (FIG. 3) may be different fromthe actions set forth in the surgical plan defined during preoperativephase 302 (FIG. 3).

There are many reasons why a surgeon may deviate from the surgical plan.For example, the surgical plan developed during preoperative phase 302may be based on incorrect estimations or assumptions regarding theanatomy of the patient. For instance, in one example, the surgical plandeveloped during preoperative phase 302 may be based on incorrectestimates of the positions or sizes of osteophytes.

In another example of why a surgeon may deviate from the surgical plandeveloped during preoperative phase 302, the surgical plan may be basedon incorrect estimates of the patient's pathology. For instance, thesurgeon may discover upon opening the patient's shoulder joint that alevel of glenoid erosion in the patient's shoulder joint is consistentwith a different category in the Walch glenohumeral osteoarthritisclassification system than was understood from previous imaging of thepatient's shoulder joint. In the case of a shoulder repair surgery, anincorrect estimate of the patient's pathology may require the surgeon toperform a reverse shoulder arthroplasty instead of a standard anatomicshoulder arthroplasty, or vice versa.

In another example of why a surgeon may deviate from the surgical plandeveloped during preoperative phase 302, the surgical plan may be basedon incorrect estimates of bone density. For instance, in this example,if bone density is too low in a particular area of a bone, such as thehumerus or scapula, the bone may not be able to adequately support animplant, such as the implant type planned during preoperative phase 302.The bone may fracture if the bone is not able to adequately support theimplant or it may be impossible to securely mount the implant onto thebone. As a result, the surgeon may be compelled to cut, drill, or reamthe bone in a manner that differs from the surgical plan to improve theseating of the implant, or use a different implant type. For instance,in one example, the initial surgical plan may call for the use of astemless humeral implant, but if the bone density of the humerus is toolow in certain areas, the surgeon may want to switch to a stemmedhumeral implant.

In other examples of why a surgeon may deviate from the surgical plandeveloped during the preoperative phase 302, the surgeon may simplyrealize that there were errors in the surgical plan. The surgeon mayalso deviate from the surgical plan inadvertently. For example, thesurgeon may cut a bone at an angle that differs somewhat from the angleindicated by the surgical plan. For instance, the surgeon may cut awaytoo much of the humeral head. In some instances, different angles of acut may result in different potential ranges of motion if notcompensated for by using different implants.

In accordance with a technique of this disclosure, a computing systemmay obtain an information model specifying a first surgical plan, whichmay be a preoperatively-defined surgical plan. The computing system maymodify the first surgical plan during intraoperative phase 306 of anorthopedic surgery to generate a second surgical plan (FIG. 3).Furthermore, in accordance with a technique of this disclosure, an XRsystem may present an XR visualization for display that is based on thesecond surgical plan. This may allow the surgeon to start using XRvisualizations to help the surgeon execute the second surgical plan.

FIG. 120 is a conceptual diagram illustrating an example system 12000 inwhich a first surgical plan is modified during intraoperative phase 306to generate a second surgical plan, in accordance with a technique ofthis disclosure. System 12000 includes a computing system 12002, an XRsystem 12004, a storage system 12006, and a network 12008. In theexample of FIG. 120, XR system 12004 includes a visualization device12010. Similar to MR system 212 of FIG. 2, XR system 12004 may alsoinclude other components, such as one or more processors and storagedevices separate from visualization device 12010. As shown in theexample of FIG. 120, visualization device 12010 may be a head-mountedvisualization device. In other examples, visualization device 12010 maybe a holographic projector or other type of device that enables a userto visualize an MR visualization. In some examples, the functions ofcomputing system 12002 are performed by one or more processors includedin visualization device 12010 or XR system 12004. In other examples,some or all functions of computing system 12002 are performed byprocessors external to visualization device 12010.

As discussed elsewhere in this disclosure, computing system 12002 maycomprise one or more computing devices operating as a system. In someexamples, computing system 12002 is part of virtual planning system 102(FIG. 1) or intraoperative guidance system 108 (FIG. 1). Computingsystem 12002 may obtain an information model that describes a surgicalplan. For example, computing system 12002 may generate the informationmodel e.g., using surgical planning software, such as BLUEPRINT™. Insome examples, computing system 12002 may obtain the information modelfrom a computer readable medium, such as a communication medium ornon-transitory computer-readable medium. Storage system 12006 may storethe information model.

FIG. 121 is a flowchart of an example operation in which a firstsurgical plan is modified during intraoperative phase 306 to generate asecond surgical plan, in accordance with a technique of this disclosure.In the example of FIG. 121, computing system 12002 may obtain aninformation model specifying a first surgical plan for an orthopedicsurgery to be performed on a patient (12100). For instance, computingsystem 12002 may generate or receive the first surgical plan asdescribed elsewhere in this disclosure. Furthermore, in the example ofFIG. 121, computing system 12002 may modify the first surgical planduring intraoperative phase 306 (FIG. 3) of the orthopedic surgery togenerate a second surgical plan (12102). Visualization device 12010 maypresent, during intraoperative phase 306 of the orthopedic surgery, avisualization for display that is based on the second surgical plan(12104). The visualization may be an XR visualization, such as an MRvisualization or a VR visualization.

In some examples, the first surgical plan is a preoperatively-definedsurgical plan for the orthopedic surgery to be performed on the patient.In other words, the first surgical plan may be defined duringpreoperative phase 302 (FIG. 3). In some examples, the first surgicalplan is an intraoperatively-defined surgical plan for the orthopedicsurgery to be performed on the patient. For instance, in one example, apreoperatively-defined surgical plan may have been modified during thecourse of the orthopedic surgery to generate the first surgical plan,which may be further modified to generate the second surgical plan.

The first surgical plan may differ from the second surgical plan invarious ways. For example, the first surgical plan may specify use of afirst surgical item, such as a surgical tool or implant. In thisexample, computing system 12002 may modify the first surgical plan suchthat the second surgical plan specifies use of a second surgical iteminstead of the first surgical item. In this example, the second surgicalitem is different from the first surgical item. For instance, the firstsurgical item and the second surgical item may be different sizes of thesame type of item or may be different types of items altogether. Forinstance, the first surgical item may be a stemless humeral implant andthe second surgical item may be a stemmed humeral implant, or viceversa. In some examples, the first surgical plan may differ from thesecond surgical plan with respect to a cutting angle or position (e.g.,a cutting angle or position of a resection of the humeral head).

In some examples, the first surgical plan may specify performance of afirst type of surgery. In this example, computing system 12002 maymodify the first surgical plan such that the second surgical planspecifies a second type of surgery that is different from the first typeof surgery. For instance, the first type of surgery and the second typeof surgery may be different ones of: a stemless standard total shoulderarthroplasty, a stemmed standard total shoulder arthroplasty, a stemlessreverse shoulder arthroplasty, a stemmed reverse shoulder arthroplasty,an augmented glenoid standard total shoulder arthroplasty, or anaugmented glenoid reverse shoulder arthroplasty. In other examples, thefirst type of surgery and the second type of surgery may be differenttypes of ankle surgery or other types of joint surgery. Althoughtechniques are described with reference to shoulder surgical procedures,similar interoperative changes to the surgical plan may also occur withregard to other types of orthopedic surgical procedures, such as for anankle arthroplasty procedure.

In some examples where the orthopedic surgery is a shoulder repairsurgery, the first surgical plan may specify a first reaming axis on aglenoid of the patient. In this example, the second surgical plan mayspecify a second reaming axis on the glenoid of the patient differentfrom the first reaming axis. For instance, the first reaming axis andthe second reaming axis may have different parameters, such as angles,entry points, and so on. In some examples where the orthopedic surgeryis a shoulder repair surgery, the first surgical plan may specify afirst reaming depth on a glenoid of the patient. In this example, thesecond surgical plan may specify a second reaming depth on the glenoidof the patient different from the first reaming depth.

Furthermore, in some examples where the orthopedic surgery is a shoulderrepair surgery, the first surgical plan may specify a first humerus cutdepth and the second surgical plan may specify a second humerus cutdepth different from the first humerus cut depth. In some examples wherethe orthopedic surgery is a shoulder repair surgery, the first surgicalplan specifies a first humerus cut angle and the second surgical planspecifies a second humerus cut angle different from the first humeruscut angle.

In some examples, the first surgical plan may specify a first implantposition and the second surgical plan specifies a second implantposition different from the first implant position. For instance, in oneexample, the first implant position and the second implant position maybe different positions at which to mount an implant to a bone. Inanother example, the first implant position and the second implantposition may be different depths of a stem of the implant, such as ahumeral implant.

Computing system 12002 may perform various actions to determine whetherand how to modify the first surgical plan to generate the secondsurgical plan. For example, computing system 12002 may generate, basedon the first surgical plan, a planned depth map that represents a3-dimensional shape that a surface of a bone of the patient will haveafter performance of a step of the orthopedic surgery if the step of theorthopedic surgery is performed according to the first surgical plan.Furthermore, computing system 12002 may generate a realized depth mapthat represents a 3-dimensional shape of the surface of the bone of thepatient after a surgeon has actually performed the step of theorthopedic surgery. In this example, computing system 12002 may useinformation provided by one or more depth cameras (e.g., depth camera(s)532 (FIG. 5)) of visualization device 12010 to generate the depth maps(e.g., the realized depth maps). Computing system 12002 may determinedifferences between the realized depth map and the planned depth map.For instance, computing system 12002 may determine differences in anglesor depths between the realized depth map and the planned depth map.Computing system 12002 may generate the second surgical plan based onthe determined differences between the realized depth map and theplanned depth map. For instance, computing system 12002 may determinethat different type of surgery or different surgical item may bepreferable given the differences between the realized depth map and theplanned depth map. In some examples, computing system 12002 uses markers(e.g., marker 3010 of FIG. 30) to determine depths and orientations ofobjects, such as bones and surgical tools.

In some examples, the orthopedic surgery is a shoulder repair surgerythat includes a step of severing a humeral head (e.g., resecting ahumeral head) with an oscillating saw tool. In such examples, a plannedcut plane is a plane along which the oscillating saw tool will cut thehumeral head if the step of the step of severing the humeral head isperformed according to the first surgical plan. Furthermore, in thisexample, a realized cut plane is a plane along which the oscillating sawtool actually cut the humeral head during performance of the step ofsevering the humeral head. In this example, computing system 12002 maydetermine differences between the planned cut plane and the realized cutplane. In addition, computing system 12002 may generate the secondsurgical plan based on the determined differences between the realizedcut plane and the planned cut plane.

In some examples where the orthopedic surgery is a shoulder repairsurgery that includes a step of severing a humeral head with anoscillating saw tool, computing system 12002 may generate a depth mapthat represents 3-dimensional shapes within a scene that includes thehumeral head and the oscillating saw tool. In such examples, computingsystem 12002 may determine the realized cut plane based on the depthmap. In some examples, a marker is fixed to the oscillating saw tool andcomputing system 12002 determines the realized cut plane based on videoimages (e.g., RGB images) of the marker captured during performance ofthe step of severing the humeral head.

Furthermore, in one example where the orthopedic surgery is a shoulderrepair surgery that includes a step of severing a humeral head with anoscillating saw tool, the first surgical plan specifies use of a firsthumeral implant. In this example, computing system 12002 may determine,based on the realized cut plane, a first range of motion that wouldresult from using the first humeral implant. Additionally, computingsystem 12002 may determine that use of a second humeral implant wouldresult in a second range of motion instead of the first range of motiongiven the realized cut plane, where the second humeral implant isdifferent from the first humeral implant. In this example, computingsystem 12002 may determine the first range of motion using a database ofcase studies that maps realized cut planes to ranges of motion, using amathematical model, or based on other algorithms. Furthermore, in thisexample, computing system 12002 may generate the second surgical plansuch that the second surgical plan specifies use of the second humeralimplant instead of the first humeral implant.

In some examples where the orthopedic surgery is a shoulder repairsurgery that includes a step of reaming a glenoid of the patient with areaming tool, a planned reaming depth is a depth to which the reamingtool will ream the glenoid if the step of reaming the glenoid isperformed according to the first surgical plan. In such examples, arealized reaming depth is a depth to which the reaming tool actuallyreamed the glenoid during performance of the step of reaming theglenoid. In one such example, computing system 12002 may determinedifferences between the planned reaming depth and the realized reamingdepth. Furthermore, in this example, computing system 12002 may generatethe second surgical plan based on the determined differences between therealized reaming depth and the planned reaming depth.

Furthermore, in one example where the orthopedic surgery is a shoulderrepair surgery that includes a step of reaming the glenoid of thepatient, computing system 12002 may generate a depth map that represents3-dimensional shapes within a scene that includes the glenoid and thereaming tool. In this example, computing system 12002 may determine therealized reaming depth based on the depth map. For instance, computingsystem 12002 may use the depth values in the depth map that correspondto the glenoid after reaming to determine how deeply the glenoid wasreamed. In some examples, a first marker is fixed to the reaming tooland a second marker is fixed to a scapula that has the glenoid. In suchexamples, computing system 12002 may determine the realized reamingdepth based on video images of the first marker and the second markercaptured during performance of the step of reaming the glenoid.

Furthermore, in one example where the orthopedic surgery is a shoulderrepair surgery that includes a step of reaming the glenoid, the firstsurgical plan may specify use of a first glenoid implant. In thisexample, computing system 12002 may determine, based on the realizedreaming depth, a first range of motion that would result from using thefirst glenoid implant. Additionally, in this example, computing system12002 may determine that use of a second glenoid implant would result ina second range of motion instead of the first range of motion given therealized reaming depth, where the second glenoid implant is differentfrom the first glenoid implant. Computing system 12002 may determine theranges of motion in accordance with any of the examples providedelsewhere in this disclosure for determining ranges of motion.Furthermore, computing system 12002 may generate the second surgicalplan such that the second surgical plan specifies use of the secondglenoid implant instead of the first glenoid implant. In some examples,the second range of motion is greater than the first range of motion.

As mentioned above, visualization device 12010 may present an XRvisualization based on the second surgical plan. The XR visualizationbased on the second surgical plan may include various types of content,such as any of the intraoperative MR content described elsewhere in thisdisclosure. For instance, the XR visualization may comprise one or morevirtual checklist items corresponding to items in a checklist of stepsof the orthopedic surgery that conforms to the second surgical plan. Inexamples where the visualization is an MR visualization, the MRvisualization may include virtual checklist items and also includeimages of real-world objects. In such examples, the images of thereal-worlds objects may be the images of the patient and operating roomas seen by the surgeon through a see-through holographic lens ofvisualization device 12010 or as seen by the surgeon on a screen ofvisualization device 12010.

In some examples, system 12000 may enable a surgeon to assess theconsequences of deviating from a surgical plan before the surgeondeviates from the surgical plan. For example, the surgeon may want toassess what the consequences would be of using a different cutting angleor depth, different reaming depth, or different surgical item thanspecified in the surgical plan e.g., because the surgeon has discoveredcertain osteophytes, different bone density than expected, or otherfactors. In some examples where system 12000 enables the surgeon toassess the consequences of deviating from the surgical plan, computingsystem 12002 may generate, during intraoperative phase 306 (FIG. 3), amodified surgical plan based on a proposed deviation from the surgicalplan. The surgeon may then have the opportunity to review the modifiedsurgical plan intraoperatively. For instance, visualization device 12010may present an XR visualization based on the modified surgical plan. Insome examples, system 12000 may determine one or more ranges of motionthat may result from deviating from the surgical plan. The ranges ofmotion may be determined based on data in a database of historicalcases, based on a mathematical model, based on a neural network model,or using another type of algorithm. An XR visualization may present theranges of motion. For instance, system 12000 may present the ranges ofmotion resulting from the modified surgical plan in the manner describedwith respect to FIG. 11A and FIG. 11B.

The surgeon may specify a proposed deviation from the surgical plan invarious ways. For instance, in an example where the surgeon is using anoscillating saw tool, the surgeon may hold the oscillating saw tool at aparticular angle with respect to a cutting plane specified by a surgicalplan for a bone. In this example, the surgeon may then provide inputthat asks system 12000 to determine the consequences of cutting the boneat the particular angle. For instance, the surgeon may provide a voicecommand saying, “what happens if I use this cutting plane instead?” Inan example where the surgeon is using drilling tool, the surgeon mayissue a voice command indicating how the surgeon proposes changing aposition or depth of a drilling hole. Similarly, where the surgeon isusing a reaming tool, the surgeon may issue a voice command indicating aproposed alternative reaming depth.

When computing system 12002 modifies a surgical plan, computing system12002 may attempt to optimize one or more surgical parameters of theorthopedic surgery given the constraints imposed by the deviation fromthe surgical plan. For example, computing system 12002 may attempt tooptimize the anticipated range of motion, optimize contact with corticalbone, optimize contact points of implants with high quality (e.g.,dense) portions of the bone, or other factors. These factors may drivepositions of implants, stem sizes of implants, surface sizes ofimplants, and other surgical parameters. Furthermore, in some examples,when computing system 12002 modifies a surgical plan, processing device8304 (FIG. 83), which may be part of computing system 12002 or anothercomputing system, may identify a next surgical item based on themodified surgical plan and present virtual information in a MRpresentation that identifies the next surgical item.

This disclosure describes to the orthopedic classification and surgeryplanning using artificial intelligence (AI) techniques such as neuralnetworks. In some examples, such AI techniques may be employed duringpreoperative phase 302 (FIG. 3) or another phase of a surgicallifecycle. Artificial neural networks (ANNs), including deep neuralnetworks (DNNs), have shown great promise as classification tools. A DNNincludes an input layer, an output layer, and one or more hidden layersbetween the input layer and the output layer. ANNs and DNNs may alsoinclude one or more other types of layers, such as pooling layers.

Each layer may include a set of artificial neurons, which are frequentlyreferred to simply as “neurons.” Each neuron in the input layer receivesan input value from an input vector. Outputs of the neurons in the inputlayer are provided as inputs to a next layer in the ANN. Each neuron ofa layer after the input layer may apply a propagation function to theoutput of one or more neurons of the previous layer to generate an inputvalue to the neuron. The neuron may then apply an activation function tothe input to compute an activation value. The neuron may then apply anoutput function to the activation value to generate an output value forthe neuron. An output vector of the ANN includes the output values ofthe output layer of the ANN.

There have been several challenges associated with application of ANNsto planning orthopedic surgery, particularly with respect to shoulderpathology. For example, some challenges relate to how to structure andtrain an ANN so that the ANN is able to provide meaningful outputregarding shoulder pathology. In another example of a challengeassociated with application of ANNs to planning orthopedic surgery,patients and healthcare professionals are understandably reluctant totrust decisions made by a computer, especially when it is unclear howthe computer made those decisions. There are therefore problems abouthow to generate output in a way that helps ensure that patients andhealthcare professionals are comfortable in trusting the output of anANN.

This disclosure describes techniques that may resolve these challengesand provide an ANN structure that provides meaningful output regardingshoulder pathology. For example, an artificial neural network (ANN),such as a DNN, has an input layer, an output layer, and one or morehidden layers between the input layer and the output layer. The inputlayer includes a plurality of input layer neurons. Each input layerneuron in the plurality of input layer neurons corresponds to adifferent input element in a plurality of input elements. The outputlayer includes a plurality of output layer neurons.

Each output layer neuron in the plurality of output layer neuronscorresponds to a different output element in a plurality of outputelements. Each output element in the plurality of output elementscorresponds to a different classification in one or more shoulderpathology classification systems. In this example, a computing systemmay generate a plurality of training datasets from past shoulder surgerycases. Each respective training dataset corresponds to a differenttraining data patient in a plurality of training data patients andcomprises a respective training input vector and a respective targetoutput vector.

For each respective training dataset, the training input vector of therespective training dataset comprises a value for each element of theplurality of input elements. For each respective training dataset, thetarget output vector of the respective training dataset comprises avalue for each element of the plurality of output elements. In thisexample, the computing system may use the plurality of training datasetsto train the neural network. Additionally, in this example, thecomputing system may obtain a current input vector that corresponds to acurrent patient. The computing system may apply the DNN to the currentinput vector to generate a current output vector. The computing systemmay then determine, based on the current output vector, a classificationof a shoulder condition of the current patient, which also may bereferred to as a shoulder classification. In some instances, theclassification is a diagnosis.

In this example, by having different output elements in the plurality ofoutput elements correspond to different classes in one or more shoulderpathology classification systems, the DNN may be able to providemeaningful output information that can be used in the classification ofshoulder conditions of patients. For instance, this may be moreefficient computationally and in terms of training time than a system inwhich different values of a neuron in the output layer correspond todifferent classes. Furthermore, in some examples, the output values ofneurons in the output layer indicate measures of confidence that theclassified shoulder condition of a patient belongs in the correspondingclass in one of the shoulder pathology classification systems. Suchconfidence values may help users consider the likelihood that thepatient may have a different class of shoulder condition than thatdetermined by the computing system using the DNN. Furthermore, it may becomputationally efficient for the output of the same output layerneurons to both express confidence levels and be used as the basis fordetermining a classification (e.g., diagnosis) of a shoulder conditionof a patient.

FIG. 122 is a block diagram illustrating an example computing system12202 that implements a DNN usable for determining a classification of ashoulder condition of the patient, in accordance with a technique ofthis disclosure. Computing system 12202 may be part of orthopedicsurgical system 100 (FIG. 1). Computing system 12202 may use the DNN toclassify the shoulder condition of the patient as part of classifying apathology in step 802 of FIG. 8. In some examples, computing system12202 includes a XR visualization device (e.g., an MR visualizationdevice or a VR visualization device) that includes one or moreprocessors that perform operations of computing system 12202.

As shown in the example of FIG. 122, system 10900 includes a computingsystem 12202, a set of one or more client devices (collectively, “clientdevices 12204”). In other examples, system 12200 may include more,fewer, or different devices and systems. In some examples, computingsystem 12202 and client devices 12204 may communicate via one or morecommunication networks, such as the Internet.

Computing system 12202 may include one or more computing devices.Computing system 12202 and client devices 12204 may include varioustypes of computing devices, such as server computers, personalcomputers, smartphones, laptop computers, and other types of computingdevices. In the example of FIG. 122, computing system 12202 includes oneor more processing circuits 12206, a data storage system 12208, and aset of one or more communication interfaces 12210A through 12210N(collectively, “communication interfaces 12210”). Data store 12208 isconfigured to store data. Communication interfaces 12210 may enablecomputing system 12202 to communicate (e.g., wirelessly or using wires)to other computing systems and devices, such as client devices 12204.For ease of explanation, this disclosure may describe actions performedby processing circuits 12206, data store 12208, and communicationinterfaces 12210 as being performed by computing system 10902 as awhole. One or more sub-systems of orthopedic surgical system 100(FIG. 1) may include computing system 12202 and client devices 12204.For example, virtual planning system 102 may include computing system12202 and client devices 12204.

Users may use client devices 12204 to access information generated bycomputing system 12202. For example, computing system 12202 maydetermine a classification of a shoulder condition of a current patient.The classification may be represented by a shoulder class among aplurality of shoulder classes in a shoulder pathology classificationsystem. In this example, users may use client devices 12204 to accessinformation regarding the classification. Because computing system 12202may be remote from client devices 12204, users of client devices 12204may consider computing system 12202 to be in a cloud-based computingsystem. In other examples, some or all the functionality of computingsystem 12202 may be performed by one or more of client devices 12204.

Computing system 12202 may implement a neural network (NN), such as aDNN. Storage system 12208 may comprise one or more computer-readabledata storage media. Storage system 12208 may store parameters of the NN.For instance, storage system 12208 may store weights of neurons of theNN, bias values of neurons of the NN, and so on.

Computing system 12202 may determine a classification of a shouldercondition of a patient based on output of the NN. In accordance with atechnique of this disclosure, output elements of the NN include outputelements corresponding to different classes in one or more shoulderpathology classification systems. The shoulder pathology classificationsystems may include a primary glenohumeral osteoarthritis classificationsystem, a rotator cuff classification system, and classification systemsfor other shoulder pathologies. For instance, the Walch classificationsystem and the Favard classification system are two different primaryglenohumeral osteoarthritis classification systems. The Warnerclassification system and the Goutalier classification system are twodifferent rotator cuff classification systems. In some examples, ashoulder pathology classification system may include classes for moregeneral categories of shoulder pathology, such as one or more of:primary glenoid humeral osteoarthritis (PGHOA), rotator cuff teararthropathy (RCTA) instability, massive rotator cuff tear (MRCT),rheumatoid arthritis, post-traumatic arthritis, and osteoarthritis.

The Walch classification system, for example, specifies five classes:1A, 1B, 2A, 2B, and 3. The Favard classification system, as anotherexample, specifies five classes: E0, E1, E2, E3, and E4. The Warnerclassification system, as a further example, specifies four classes ofrotator cuff atrophy: none, mild, moderate, and severe. The Goutalierclassification system, as a further example, specifies five classes: 0(completely normal muscle), I (some fatty streaks), II (amount of muscleis greater than fatty infiltration), III (amount of muscle is equal tofatty infiltration), IV (amount of fatty infiltration is greater thanmuscle).

In some examples, computing system 12202 may determine theclassification of the shoulder condition of the patient based on acomparison of the values of the output elements generated by the NN. Forexample, the values of the output elements may correspond to confidencevalues that indicate levels of confidence that the patient's shouldercondition belongs in the classes that correspond to the output layerneurons that generated the values. For instance, the values of theoutput elements may be the confidence values or computing system 12202may calculate the confidence values based on the values of the outputelements.

In some examples, the output function of the output layer neuronsgenerates the confidence values. Furthermore, computing system 12202 mayidentify which of the confidence values is highest. In this example,computing system 12202 may determine that the shoulder pathology classcorresponding to the highest confidence value is the classification ofthe shoulder condition of the current patient. In some examples, if noneof the confidence values is above a threshold, computing system 12202may generate output indicating that computing system 12202 is unable tomake a definitive classification.

As mentioned above, in some examples, the output elements of the NNinclude confidence values. In one such example, a confidence valuefunction outputs confidence values. The confidence value function may bethe output function of the output layer neurons of the NN. In thisexample, all possible confidence values output by the confidence valuefunction are within a predefined range. Furthermore, in this example,computing system 12202 may apply the NN to an input vector to generatean output vector. As part of applying the NN, computing system 12202may, for each respective output layer neuron in the plurality of outputlayer neurons, calculate an output value of the respective output layerneuron.

Computing system 12202 may then apply the confidence value function withthe output value of the respective output layer neuron as input to theconfidence value function. The confidence value function outputs aconfidence value for the respective output layer neuron. In thisexample, for each respective output layer neuron in the plurality ofoutput layer neurons, the output element corresponding to the respectiveoutput layer neuron specifies the confidence value for the respectiveoutput layer neuron. Furthermore, for each respective output layerneuron in the plurality of output layer neurons, the confidence valuefor the respective output layer neuron is a measure of confidence thatthe shoulder condition of the current patient belongs to a class in theone or more shoulder pathology classification systems that correspondsto the output element corresponding to the respective output layerneuron.

Computing system 12202 may use various confidence value functions. Forexample, computing system 12202 may apply a hyperbolic tangent function,a sigmoid function, or another type of function that output values thatare within a predefined range. The hyperbolic tangent function (tanh)has the form γ(c)=tanh(c)=(e^(c)−e^(−c))/(e^(c)+e^(−c)). The hyperbolictangent function takes real-valued arguments, such as output values ofoutput layer neurons, and transforms them to the range (−1, 1). Thesigmoid function has the form γ(c)=1/(1+e^(−c)). The sigmoid functiontakes real-valued arguments, such as output values of output layerneurons, and transforms them to the range (0, 1).

Computing system 12202 may use a plurality of training datasets to trainthe NN. Each respective training dataset may correspond to a differenttraining data patient in a plurality of previously classified trainingdata patients. For instance, a first training dataset may correspond toa first training data patient, a second training dataset may correspondto a second training data patient, and so on. A training dataset maycorrespond to a training data patient in the sense that the trainingdataset may include information regarding the patient. The training datapatients may be real patients who have classified shoulder conditions.In some examples, the training data patients may include simulatedpatients.

Each respective training dataset may include a respective training inputvector and a respective target output vector. For each respectivetraining dataset, the training input vector of the respective trainingdataset comprises a value for each element of the plurality of inputelements. In other words, the training input vector may include a valuefor each input layer neuron of the NN. For each respective trainingdataset, the target output vector of the respective training dataset maycomprise a value for each element of the plurality of output elements.In other words, the target output vector may include a value for eachoutput layer neuron of the NN.

In some examples, the values in the target output vector are based onconfidence values. Such confidence values may, in turn, be based onlevels of confidence expressed by one or more trained healthcareprofessionals, such as orthopedic surgeons. For instance, a trainedhealthcare professional may be given the information in the traininginput vector of a training dataset (or information from which thetraining input vector of the training dataset is derived) and may beasked to provide a confidence level that the training data patient has ashoulder condition belonging to each class in each of the shoulderpathology classification systems.

For instance, in an example where the shoulder pathology classificationsystems include the Walch classification system, the healthcareprofessional may indicate that her level of confidence that the trainingdata patient's shoulder condition belongs to class A1 is 0 (meaning shedoes not at all believe that the training data patient's shouldercondition belongs to class A1), indicate that her level of confidencethat the training data patient's shoulder condition belongs to class A2is 0; indicate that her level of confidence that the training datapatient's shoulder condition belongs to class B1 is 0.75 (meaning she isfairly confident that the training data patient's shoulder conditionbelongs to class B1); indicate that her level of confidence that thetraining data patient's shoulder condition belongs to class B2 is 0.25(meaning she believes that there is a smaller chance that the trainingdata patient's shoulder condition belongs to class B2); and may indicatethat her level of confidence that the training data patient's shouldercondition belongs to class C is 0. In some examples, computing system12202 may apply the inverse of the confidence value function to theconfidence values provided by the healthcare professional to generatevalues to include in the target output vector. In some examples, theconfidence values provided by the healthcare professional are the valuesincluded in the target output vector.

Different healthcare professionals may have different levels ofconfidence that the same training data patient has a shoulder conditionbelonging to each class in each of the shoulder pathology classificationsystems. Hence, in some examples, the confidence values upon which thevalues in the target output vector are based may be averages orotherwise determined from the confidence levels provided by multiplehealthcare professionals.

In some such examples, the confidence levels of some healthcareprofessionals may be given greater weight in a weighted average ofconfidence levels than the confidence levels of other healthcareprofessionals. For instance, the confidence levels of a preeminentorthopedic surgeon may be given greater weight than the confidencelevels of other orthopedic surgeons. In another example, the confidencelevels of healthcare professionals or training data patients inparticular regions or hospitals may be given greater weight thanhealthcare professionals or training data patients from other regions orhospitals. Advantageously, such weighted averaging may allow the NN tobe tuned according to various criteria and preferences.

For instance, a healthcare professional may prefer to use a NN that hasbeen trained such that confidence levels are weighted in particularways. In some examples where training datasets include training datasetsbased on a healthcare professional's own cases, the healthcareprofessional (e.g., an orthopedic surgeon) may prefer to use a NNtrained using training datasets where the healthcare professional's owncases are weighted more heavily or exclusively using the healthcareprofessional's own cases. In this way, the NN may generate outputtailored to the healthcare professional's own style of practice.Moreover, as mentioned above, healthcare professionals and patients mayhave difficulty trusting the output of a computing system. Accordingly,in some examples, computing system 12202 may provide informationindicating that the NN was trained to emulate the decisions of thehealthcare professionals themselves and/or particularly trustedorthopedic surgeons.

In some examples, the confidence levels of different healthcareprofessionals for the same training data patient may be used ingenerating different training datasets. For instance, the confidencelevels of a first healthcare professional with respect to a particulartraining data patient may be used to generate a first training datasetand the confidence levels of a second healthcare professional withrespect to the same training data patient may be used to generate asecond training dataset.

Furthermore, in some examples, computing system 12202 may provideconfidence values for output to one or more users. For instance,computing system 12202 may provide the confidence values to clientdevices 12204 for display to one or more users. In this way, the one ormore users may be better able to understand how computing system 12202may have arrived at the classification of the shoulder condition of apatient.

In some examples, to expand the universe of training datasets, computingsystem 12202 may automatically generate confidence values fromelectronic medical records. For instance, in one example, an electronicmedical record for a patient may include data from which computingsystem 12202 may form an input vector and may include data indicating asurgeon's classification of a patient's shoulder condition. In thisexample, computing system 12202 may infer a default level of confidencefrom the classification. The default level of confidence may havevarious values (e.g., 0.75, 0.8, etc.). While such a default level ofconfidence may not reflect the surgeon's actual level of confidence,imputing a level of confidence may be help increase the number ofavailable training datasets, which may improve the accuracy of the NN.

In some examples, the training datasets are weighted based on healthoutcomes of the training data patients. For example, a training datasetmay be given higher weight if the training data patient associated withthe training dataset had all positive health outcomes. However, atraining dataset may be given a lower weight if the associated trainingdata patient had less positive health outcomes. During training,computing system 12202 may use a loss function that weights the trainingdatasets based on the weights given to the training datasets.

In some examples, as part of generating the training datasets, computingsystem 12202 may select the plurality of training datasets from adatabase of training datasets based on one or more training datasetselection criteria. In other words, computing system 12202 may excludecertain training datasets from the training process of the NN if thetraining datasets do not satisfy the training dataset selectioncriteria. In the example of FIG. 122, data storage system 12208 stores adatabase 12212 that contains training datasets from past shouldersurgery cases.

There may be a wide variety of training dataset selection criteria. Forinstance, in one example, the one or more training data set selectioncriteria may include which surgeon operated on the plurality of trainingdata patients. In some examples, the one or more training datasetselection criteria include a region in which the training data patientslive. In some examples, the one or more training dataset selectioncriteria include a region associated with one or more surgeons (e.g., aregion in which the one or more surgeons practice, live, were licensed,were trained, etc.).

In some examples, the one or more training dataset selection criteriainclude postoperative health outcomes of the training data patients. Insuch examples, the postoperative health outcomes of the training datapatients may include one or more of: postoperative range of motion,presence of postoperative infection, or postoperative pain. Thus, insuch examples, the training datasets upon which the NN is trained mayexclude training datasets where adverse health outcomes occurred.

Additional training datasets may be added to the database over time andcomputing system 12202 may use the additional training datasets to trainthe NN. Thus, the NN may continue to improve over time as more trainingdatasets are added to the database.

Computing system 12202 may apply one of various techniques to use thetraining datasets to train the NN. For example, computing system 12202may use one of the various standard backpropagation algorithms known inthe art. For instance, as part of training the NN, computing system12202 may apply a cost function to determine cost values based ondifferences between the output vector generated by the NN and the targetoutput vector. Computing system 12202 may then use the cost values in abackpropagation algorithm to update the weights of neurons in the NN.

FIG. 123 illustrates an example NN 12300 that may be implemented bycomputing system 12202 with the system of FIG. 122. In the example ofFIG. 123, NN 12300 includes an input layer 12302, an output layer 12304,and one or more hidden layers 12306 between input layer 12302 and outputlayer 12304. In the example of FIG. 123, neurons are represented ascircles. Although in the example of FIG. 123, each layer is shown asincluding six neurons, layers in NN 12300 may include more or fewerneurons. Furthermore, although NN 12300 is shown in FIG. 123 as being afully connected network, NN 12300 may have a different architecture. Forinstance, NN 12300 may not be a fully connected network, may have one ormore convolutional layers, or may otherwise have a differentarchitecture from that shown in FIG. 123.

In some implementations, NN 12300 can be or include one or moreartificial neural networks (also referred to simply as neural networks).A neural network can include a group of connected nodes, which also canbe referred to as neurons or perceptrons. A neural network can beorganized into one or more layers. Neural networks that include multiplelayers can be referred to as “deep” networks. A deep network can includean input layer, an output layer, and one or more hidden layerspositioned between the input layer and the output layer. The nodes ofthe neural network can be connected or non-fully connected.

NN 12300 can be or include one or more feed forward neural networks. Infeed forward networks, the connections between nodes do not form acycle. For example, each connection can connect a node from an earlierlayer to a node from a later layer.

In some instances, NN 12300 can be or include one or more recurrentneural networks. In some instances, at least some of the nodes of arecurrent neural network can form a cycle. Recurrent neural networks canbe especially useful for processing input data that is sequential innature. In particular, in some instances, a recurrent neural network canpass or retain information from a previous portion of the input datasequence to a subsequent portion of the input data sequence through theuse of recurrent or directed cyclical node connections.

In some examples, sequential input data can include time-series data(e.g., sensor data versus time or imagery captured at different times).For example, a recurrent neural network can analyze sensor data versustime to detect or predict a swipe direction, to perform handwritingrecognition, etc. Sequential input data may include words in a sentence(e.g., for natural language processing, speech detection or processing,etc.); notes in a musical composition; sequential actions taken by auser (e.g., to detect or predict sequential application usage);sequential object states; etc. Example recurrent neural networks includelong short-term (LSTM) recurrent neural networks; gated recurrent units;bi-direction recurrent neural networks; continuous time recurrent neuralnetworks; neural history compressors; echo state networks; Elmannetworks; Jordan networks; recursive neural networks; Hopfield networks;fully recurrent networks; sequence-to-sequence configurations; etc.

In some implementations, NN 12300 can be or include one or moreconvolutional neural networks. In some instances, a convolutional neuralnetwork can include one or more convolutional layers that performconvolutions over input data using learned filters. Filters can also bereferred to as kernels. Convolutional neural networks can be especiallyuseful for vision problems such as when the input data includes imagerysuch as still images or video. However, convolutional neural networkscan also be applied for natural language processing.

NN 12300 may be or include one or more other forms of artificial neuralnetworks such as, for example, deep Boltzmann machines; deep beliefnetworks; stacked autoencoders; etc. Any of the neural networksdescribed herein can be combined (e.g., stacked) to form more complexnetworks.

In the example of FIG. 123, an input vector 12308 includes a pluralityof input elements. Each of the input elements may be a numerical value.Input layer 12302 includes a plurality of input layer neurons. Eachinput layer neuron in the plurality of input layer neurons included ininput layer 12302 may correspond to a different input element in aplurality of input elements. In other words, input layer 12302 mayinclude a different neuron for each input element in input vector 12308.

Furthermore, in the example of FIG. 123, an output vector 12310 includesa plurality of output elements. Each of the output elements may be anumerical value. Output layer 12304 includes a plurality of output layerneurons. Each output layer neuron in the plurality of output layerneurons corresponds to a different output element in the plurality ofoutput elements. In other words, each output layer neuron in outputlayer 12304 corresponds to a different output element in output vector12310.

Input vector 12308 may include a wide variety of information. Forexample, input vector 12308 may include morphological measurements ofthe patient. In some examples where input vector 12308 includesmeasurements of the patient's morphology, input vector 12308 maydetermine the measurements based on medical images of the patient, suchas CT images, MRI images, or other types of images. For instance,computing system 12202 may obtain the medical images of a currentpatient. For instance, computing system 12202 may obtain the medicalimages from an imaging machine (e.g., a CT machine, MRI machine, orother type of imaging machine), an electronic medical record of thepatient, or another data source. In this example, computing system 12202may segment the medical images to identify internal structures of thecurrent patient, such as soft tissue and bone. For instance, in oneexample, computing system 12202 may apply an artificial neural networktrained to identify portions of medical images that correspond to bonesor soft tissue. Furthermore, in this example, computing system 12202 maydetermine the plurality of measurements based on relative positions ofthe identified internal structures of the current patient. In thisexample, the plurality of input elements may include an input elementfor each measurement in the plurality of measurements.

As mentioned elsewhere in this disclosure, computing system 12202 mayinclude one or more computing devices. Hence, various functions ofcomputing system 12202 may be performed by various combinations of thecomputing devices of computing system 12202. For instance, in someexamples, a first computing device of computing system 12202 may segmentthe images, a second computing device of computing system 12202 maytrain the DNN, a third computing device of computing system 12202 mayapply the DNN, and so on. In other examples, a single computing deviceof computing system 12202 may segment the images, train the DNN, andapply the DNN.

As mentioned above, computing system 12202 may determine a plurality ofmeasurements of morphological characteristics of the patient. Suchmeasurements may include distance measurements, angle measurements, andother types of numerical characterizations of measurable relationshipsof and/or between structures of a patient. For example, the measurementsmay include any combination of values relating to one or more of:

-   -   a glenoid version: an angular orientation of an axis of the        glenoid articular surface relative to a transverse axis of the        scapula.    -   a glenoid inclination: the superior/inferior tile of the glenoid        relative to the scapula.    -   a glenoid orientation/direction: the 3-dimensional orientation        of the glenoid in a 3-dimensional space.    -   a glenoid best fit sphere radius: a radius of a best-fit sphere        for the patient's glenoid. The best-fit sphere is a conceptual        sphere that is sized such that a sector of the sphere would sit        flush as possible with the patient's glenoid.    -   a glenoid best fit sphere root mean square error: the mean        square error of the difference between the patient's glenoid and        the sector of the best-fit sphere.    -   a reverse shoulder angle: the tilt of the inferior part of the        glenoid.    -   a critical shoulder angle: the angle between the plane of the        glenoid fossa and the connecting line to the most inferolateral        point of the acromion.    -   acromion humeral space: the space between the acromion and the        top of the humerus.    -   glenoid humeral space: the space between the glenoid and the        humerus.    -   humeral version: the angle between the humeral orientation and        the epicondylar axis.    -   humeral neck shaft angle: the angle between the humeral anatomic        neck normal vector and the intramedullary axis.    -   humeral head best fit sphere radius and root mean square error:        a radius of a best-fit sphere for the head of the patient's        humerus. The best-fit sphere is a conceptual sphere that is        sized such that a sector of the sphere matches the surface of        the humeral head as much as possible. The root mean square error        indicates the error between the best-fit sphere and the        patient's actual humeral head.    -   humeral subluxation: a measure of the subluxation of the humerus        relative to the glenoid.    -   humeral orientation/direction: the orientation of the humeral        head in a 3-dimensional space.    -   a measurement of an epiphysis of the patient's humerus,    -   a measurement of a metaphysis of the patient's humerus,    -   a measurement of a diaphysis of the patient's humerus,    -   retroversion of a bone

In some examples, input vector 12308 may include information (e.g., incombination with zero or more other example types of input datadescribed herein) based on a rotator cuff assessment of the patient. Forinstance, input vector 12308 may include information, alone or incombination with morphological inputs described above, regarding fattyinfiltration of the rotator cuff, atrophy of the rotator cuff, and/orother information about the patient's rotator cuff. In some examples,fatty infiltration measures and atrophy measures for soft tissue used asinputs to the neural network may be derived, for example, by any of thesoft tissue modeling techniques as described in this application. Insome examples, the information regarding the patient's rotator cuff maybe expressed in terms of a class in a shoulder pathology classificationsystem, such as a Warner classification system or a Goutalierclassification system.

In some examples, input vector 12308 may include (e.g., in combinationwith zero or more other example types of input data described herein)patient range of motion information. In some examples, the patient rangeof motion information may be generated using a motion tracking device,as described elsewhere in this disclosure.

Furthermore, in some examples, input vector 12308 may includeinformation (e.g., in combination with zero or more other example typesof input data described herein) that specifies a class in one or moreshoulder pathology classification systems. In such examples, the outputvector may include output elements corresponding to classes in one ormore different shoulder pathology classification systems. For example,input vector 12308 may include information that specifies a class in arotator cuff classification system and output vector 12310 may includeoutput elements corresponding to classes in a glenohumeralosteoarthritis classification system.

In some examples, input vector 12308 may include information (e.g., incombination with zero or more other example types of input datadescribed herein, including morphological inputs and/or rotator cuffinputs) that specifies bone density scores for humerus and/or glenoid.Other information included in input vector 12308 may include demographicinformation, such as patient age, patient activities, patient gender,patient body mass index (BMI), and so on. In some examples, input vector12308 may include information regarding the speed of onset of thesymptoms (e.g., gradual or sudden). The plurality of input elements ininput vector 12308 also may include patient objectives for participationin activities such as particular exercises/sport types, ranges ofmotion, etc.

FIG. 124 is a flowchart illustrating an example operation of computingsystem 12202 in using a DNN to determine a classification of a shouldercondition, in accordance with a technique of this disclosure. In theexample of FIG. 124, computing system 12202 generates a plurality oftraining datasets from past shoulder surgery cases (12400). In someexamples, generating the plurality of training datasets comprisesretrieving selected training datasets from a database of historicalshoulder cases. In the example of FIG. 124, a NN, such as NN 12300 (FIG.123) has an input layer, an output layer, and one or more hidden layersbetween the input layer and the output layer. The input layer includes aplurality of input layer neurons. Each input layer neuron in theplurality of input layer neurons corresponds to a different inputelement in a plurality of input elements. The output layer includes aplurality of output layer neurons. Each output layer neuron in theplurality of output layer neurons corresponds to a different outputelement in a plurality of output elements. Each output element in theplurality of output elements corresponds to a different class in one ormore shoulder pathology classification systems.

Each respective training dataset corresponds to a different trainingdata patient in a plurality of training data patients from past shouldersurgery cases and comprises a respective training input vector and arespective target output vector. For each respective training dataset,the training input vector of the respective training dataset comprises avalue for each element of the plurality of input elements. For eachrespective training dataset, the target output vector of the respectivetraining dataset comprises a value for each element of the plurality ofoutput elements.

Furthermore, in the example of FIG. 124, computing system 12202 uses theplurality of training datasets to train the NN (12402). Additionally,computing system 12202 may obtain a current input vector thatcorresponds to a current patient (12404), e.g., including a combinationof various patient inputs as described above. Computing system 12202 mayapply the NN to the current input vector to generate a current outputvector (12406). For instance, computing system 12202 may provide thecurrent input vector to the input layer of the NN and may performforward propagation through the NN to generate the current outputvector.

Furthermore, computing system 12202 may determine, based on the currentoutput vector, a classification of a shoulder condition of the currentpatient (12408). In some examples, the classification of the shouldercondition of the current patient may be a diagnosis. As discussed above,the shoulder condition may be expressed by a classification from aselected shoulder classification system. In addition, shoulder conditionmay be accompanied by a confidence measure, value or other indication.Computing system 12202 may perform these actions in accordance with anyof the examples or combination of examples provided elsewhere in thisdisclosure.

Healthcare professionals may use the classification of the shouldercondition in various ways. For example, a surgeon may use theclassification to select a type of surgery to perform on the patient. Insome examples, the surgeon may use the classification to select surgicalitems (e.g., implants, tools, etc.) to use to perform a surgery on thepatient. In some examples, a physical therapist or other healthcareprofessional may use the classification to determine a rehabilitationplan for the patient.

In an example, the disclosure relates to the orthopedic surgeryplanning, including surgical procedure type selection, using artificialintelligence techniques such as neural networks. Prior research has notresolved questions regarding how to structure and train a NN so that theNN is able to provide meaningful output regarding shoulder pathology,prior research has not resolved questions regarding how to structure andtrain a NN so that the NN is able to provide meaningful output regardingwhich type of shoulder surgery to perform on a patient. Example types ofshoulder surgeries may include a standard total shoulder arthroplastyand a reverse shoulder arthroplasty. In another example of a challengeassociated with application of NNs to planning orthopedic surgery,patients and healthcare professionals are understandably reluctant totrust decisions made by a computer, especially when it is unclear howthe computer made those decisions. There are therefore problems abouthow to generate output in a way that helps ensure that patients andhealthcare professionals are comfortable in trusting the output of a NN.

This disclosure describes techniques that may resolve these challengesand may provide a NN structure that provides meaningful output regardinga recommended type of shoulder surgery for a patient. In accordance witha technique of this disclosure, a NN (e.g., a DNN) may have an inputlayer, an output layer, and one or more hidden layers between the inputlayer and the output layer. The input layer includes a plurality ofinput layer neurons. Each input layer neuron in the plurality of inputlayer neurons corresponds to a different input element in a plurality ofinput elements. The output layer includes a plurality of output layerneurons. Each output layer neuron in the plurality of output layerneurons corresponds to a different output element in a plurality ofoutput elements. The plurality of output elements includes a pluralityof surgery type output elements.

In accordance with a technique of this disclosure, each surgery typeoutput element in the plurality of surgery type output elementscorresponds to a different type of shoulder surgery in a plurality oftypes of shoulder surgery. Furthermore, in accordance with thistechnique, a computing system may generate a plurality of trainingdatasets. Each respective training dataset corresponds to a differenttraining data patient in a plurality of training data patients andcomprises a respective training input vector and a respective targetoutput vector. For each respective training dataset, the training inputvector of the respective training dataset comprises a value for eachelement of the plurality of input elements. For each respective trainingdataset, the target output vector of the respective training datasetcomprises a value for each element of the plurality of output elements.The computing system may use the plurality of training datasets to trainthe NN.

Additionally, the computing system may obtain a current input vectorthat corresponds to a current patient. The computing system may applythe NN to the current input vector to generate a current output vector.Additionally, the computing system may determine, based on the currentoutput vector, a recommended type of shoulder surgery for the currentpatient. The computing system may use the NN to determine therecommended type of shoulder surgery for the current patient as part ofselecting a surgical plan for the patient in step 804 of FIG. 8.

In this example, by having different output elements in the plurality ofoutput elements correspond to different shoulder types of shouldersurgeries, the NN may be able to provide meaningful output informationthat can be used in the selection of shoulder surgery procedure types tobe used to treat shoulder conditions of patients. Furthermore, in someexamples, the output values of neurons in the output layer indicatemeasures of confidence that the patient should undergo a particular typeof shoulder surgery. Such confidence values may help users consider thelikelihood that the patient should instead undergo a different type ofshoulder surgery. Furthermore, it may be computationally efficient forthe output of the same output layer neurons to both express confidencelevels and be used as the basis for recommending a particular type ofshoulder surgery for a patient.

FIG. 125 is a block diagram illustrating example functional componentsof a computing system 12500 for using a NN to determine a recommendedsurgery for a shoulder condition, in accordance with a technique of thisdisclosure. Computing system 12500 may be implemented in the same way ascomputing system 12202 (FIG. 122) and may form part of orthopedicsurgical system 100 (FIG. 1). In some examples, the same computingsystem performs the roles of both computing system 12202 and computingsystem 12500. In some examples, computing system 12500 includes an XRvisualization device (e.g., an MR visualization device or a VRvisualization device) that includes one or more processors that performoperations of computing system 12500.

In the example of FIG. 125, computing system 12500 includes a NN 12502and storage system 12504. Storage system 12504 may store a database12506. NN 12502 may be implemented in a manner similar to NN 12300 ofFIG. 123. That is, NN 12502 may include an input layer, an output layer,and one or more hidden layers between the input layer and the outputlayer. Moreover, each input layer neuron in the plurality of input layerneurons of the input layer of NN 12502 may correspond to a differentinput element in a plurality of input elements in an input vector. Eachoutput layer neuron in the plurality of output layer neurons in theoutput layer of NN 12502 may correspond to a different output element ina plurality of output elements in an output vector. However, NN 12502may have different parameters, such as different numbers of hiddenlayers, different numbers of neurons in the input layer, differentnumbers of neurons in the output layer, and so on.

Storage system 12504 may comprise one or more computer-readable datastorage media. Storage system 12504 may store parameters of NN 12502.For instance, storage system 12504 may store weights of neurons of NN12502, bias values of neurons of NN 12502, and so on. Like database12212, database 12506 may contain training datasets from past shouldersurgery cases

Computing system 12500 may apply NN 12502 to an input vector to generatean output vector. For example, the output layer of NN 12502 may include(and in some examples be limited to) a plurality of output layerneurons. Each of the output layer neurons may correspond to a differentoutput element in a plurality of surgery type output elements. Theoutput vector may include the plurality of surgery type output elements.Each of the surgery type output elements may correspond to a differenttype of shoulder surgery. Example types of shoulder surgery types, whichmay be presented as outputs, may include a stemless standard totalshoulder arthroplasty, a stemmed standard total shoulder arthroplasty, astemless reverse shoulder arthroplasty, a stemmed reverse shoulderarthroplasty, an augmented glenoid standard total shoulder arthroplasty,an augmented glenoid reverse shoulder arthroplasty, and other types oforthopedic shoulder surgery. A shoulder surgery may be “standard” in thesense that, after surgery, the patient's shoulder joint has the standardanatomical configuration where the scapula side of the shoulder jointhas a concave surface and the humerus side of the shoulder surgery has aconvex surface. A “reverse” shoulder surgery on the other hand resultsin the opposite configuration where a convex surface is attached to thescapula and a concave surface is attached to the humerus. In someexamples, the types of shoulder surgery types include a plurality ofrevision surgeries on a patient's shoulder. Examples of revisionsurgeries on the patient's shoulder include . . .

Additionally, computing system 12500 may determine a recommended type ofshoulder surgery for a patient based on the current output vector. Forexample, computing system 12500 may determine which output element inthe output vector corresponds to the type of shoulder surgery with thegreatest confidence value.

The input vector for NN 12502 may include some, a combination of, or allof the input elements described above with respect to computing system12202 and NN 12300. For example, the input elements may include one ormore of measurements of morphological characteristics of a patient(including soft tissue modeling and bone modeling), demographicinformation regarding the patient, range of motion information regardingthe patient, and so on. Furthermore, the training datasets used intraining NN 12502 may be selected and generated in accordance with theexamples provided above with respect to the NN for classifying apatient's shoulder condition. For instance, the training datasets may beselected based on one or more training dataset selection criteria. Asdescribed elsewhere in this disclosure, such training dataset selectioncriteria may include which surgeon operated on the plurality of trainingdata patients, a region in which the training data patients live, aregion associated with one or more surgeons, postoperative healthoutcomes of the training data patients, and so on. NN 12502 may betrained in a manner similar to the examples provided above with respectto the NN for classifying a patient's shoulder condition. For instance,the target output vectors of training datasets NN 12502 may includeoutput elements indicating confidence levels that a particular type ofshoulder surgery should be performed on a patient.

FIG. 126 is a flowchart illustrating an example operation of a computingsystem that uses a NN to determine a recommended type of shouldersurgery for a patient, in accordance with a technique of thisdisclosure. In the example of FIG. 126, computing system 12500 generatesa plurality of training datasets (12600). In this example, a NN has aninput layer, an output layer, and one or more hidden layers between theinput layer and the output layer. The input layer includes a pluralityof input layer neurons. Each input layer neuron in the plurality ofinput layer neurons corresponding to a different input element in aplurality of input elements. The output layer includes a plurality ofoutput layer neurons.

Each output layer neuron in the plurality of output layer neuronscorresponding to a different output element in a plurality of outputelements. The plurality of output elements includes a plurality ofsurgery type output elements. Each surgery type output element in theplurality of surgery type output elements corresponds to a differenttype of shoulder surgery in a plurality of types of shoulder surgery.Each respective training dataset corresponds to a different trainingdata patient in a plurality of training data patients and comprises arespective training input vector and a respective target output vector.For each respective training dataset, the training input vector of therespective training dataset comprises a value for each element of theplurality of input elements. For each respective training dataset, thetarget output vector of the respective training dataset comprises avalue for each element of the plurality of output elements.

Furthermore, in the example of FIG. 126, computing system 12500 uses theplurality of training datasets to train the NN (12602). Additionally,computing system 12500 may obtain a current input vector thatcorresponds to a current patient (12604). Computing system 12500 mayapply the NN to the current input vector to generate a current outputvector (12606). Computing system 12500 may determine, based on thecurrent output vector, a recommended type of shoulder surgery for thecurrent patient (12608). Computing system 12500 may perform theseactivities in accordance with the examples provided elsewhere in thisdisclosure.

Although MR system 212 has been described above in the context of asystem for use in planning and implementing surgical repair proceduresfor the shoulder, applications of MR system 212 are not so limited. Forexample, MR system 212 and Augmented Surgery Mode can be used tofacilitate replacement of other joints or repair or reconstruction ofother bones. Thus, any of the features described above can be used toassist with any orthopedic surgical procedures, including but notlimited to procedures involving the elbow, ankle, knee, hip, or foot. Itshould also be understood that MR system 212 and its features can beused in surgical procedures in which external fixation devices are usedto fixate or repair bone, such as a Charcot procedure involving thefoot. Furthermore, MR system 212 and its features are not limited toapplications involving only bone structures. As an example, becausecertain types of image data, such as CT images and MRI images, canprovide information about soft tissue structures, including the skin, MRsystem 212 can be used to visualize and locate an optimal incisionlocation and thus can further enhance minimally invasive surgeries.

The use of MR system 212 and its augmented surgery features are notlimited to use by surgeons or other care providers. As an example, theinformation generated by MR system 212, including registration andtracking, can be used to control robotic arms that may be present in anoperating environment.

MR system 212 can be one component of an advanced surgical system forenhancing surgical outcomes. In addition to the virtual preplanning andthe use of mixed reality to perform the surgery, the implant can includevarious sensors to provide information after the surgery, as well astransceivers (e.g., RF transceivers) that facilitate collection of thedata gathered by the sensors. Such data can be used to, for example,monitor the patient's recovery and assist with the patient's recovery(e.g., by prompting the patient to move the joint, such as via anapplication installed on a mobile device used by the patient, as oneexample). The data gathered by the sensors also can be input into adatabase where it can be used by surgeons or artificial intelligencesystems to assist with planning future surgical cases.

Although many of examples of this disclosure have been provided withrespect to shoulder joint repair surgery, many techniques of thisdisclosure are applicable to other types of orthopedic surgery. Forexample, many techniques of this disclosure may be applicable to anklesurgery (e.g., total ankle arthroplasty). In the example of a totalankle arthroplasty, a surgeon may perform a distal tibial cut, aproximal calcaneus cut, and two other medial/lateral cuts. To do so, thesurgeon may need to place a cutting guide on the ankle joint. Thecutting guide is placed so that the cuts will be perpendicular to themechanical axis of the tibia. The placement of the cutting guide is thenrefined by adjusting three angles relative to the three anatomicalplanes (axial, sagittal and coronal). The surgeon can perform these cutsusing a cut jig or can perform these cuts directly using an oscillatingsaw. Next, the surgeon performs the posterior and anterior talar chamfercut.

Many of the examples provided above with regards to cutting and drillingare applicable to the cutting and drilling operations performed during atotal ankle arthroplasty. For example, during preoperative phase 302(FIG. 3) and intraoperative phase 306 (FIG. 3), orthopedic surgicalsystem 100 (FIG. 1) may provide XR visualizations (e.g., MRvisualizations or VR visualizations) that include patient-specificvirtual 3D models of a patient's ankle anatomy. This may help surgeonsplan and perform total ankle arthroplasties.

Furthermore, during the intraoperative phase 306 (FIG. 3) of a totalankle arthroplasty, visualization device 213 of MR system 212 maypresent an MR visualization that includes virtual guidance, such asvirtual cutting planes, virtual drilling axes, and virtual entry pointsthat help the surgeon perform precise cuts, drill holes, and position orplace prosthetic components. For instance, the MR visualization mayinclude cutting planes for the distal tibial cut, the proximal calcaneuscut, and so on. Prosthetic implant components for ankle arthroplasty mayinclude, in one example, a talar dome, a tibial tray, and associatedpegs or other anchor components. Moreover, a registration processsimilar to that described elsewhere in this disclosure with respect toshoulder repair surgery may be used in the context of total anklearthroplasty. For instance, instead of using a center of a glenoid as alandmark for aligning a virtual 3D model with the patient's realanatomy, another landmark (e.g., the bottom of the tibia) on thepatient's ankle may be used.

FIG. 150 is a flowchart illustrating example stages of an ankle jointrepair surgery. The surgeon may wear or otherwise use a visualizationdevice, such as visualization device 213, during some or all of thesteps of the surgical process of FIG. 150. In other examples, an anklesurgery may include more, fewer, or different steps. For example, anankle surgery may include steps for adding cement, and/or other steps.In some examples, visualization device 213 may present virtual guidanceto guide the surgeon, nurse, or other users through the steps in thesurgical workflow.

In the example of FIG. 150, a surgeon performs an incision process(15002). During the incision process, the surgeon makes a series ofincisions to expose a patient's ankle joint (e.g., to expose at least aportion of the patient's tibia and at least a portion of the patient'stalus). In some examples, an MR system (e.g., MR system 212, MR system1800A, etc.) may help the surgeon perform the incision process, e.g., bydisplaying virtual guidance imagery illustrating how and/or where tomake the incision. As discussed above, MR system 212 may display avirtual checklist, with each item on the checklist items correspondingto an item in a checklist of steps of an orthopedic surgery. Forinstance, MR system 212 may display a virtual checklist having achecklist item specifying a current step of performing an incisionprocess.

The surgeon may perform a registration process that registers a virtualtibia object with the patient's actual tibia bone (15004) in the fieldof view presented to the surgeon by visualization device 213. Forinstance, MR system 212 may obtain the virtual tibia object from storagesystem 206 of FIG. 2. Similar to the virtual glenoid object discussedabove, the virtual tibia object may be generated based on pre-operativeimaging (e.g., CT imaging) of the patient's tibia. MR system 212 mayperform the registration using any suitable process. For instance, MRsystem 212 may perform the registration of the virtual tibia object withthe patient's actual tibia bone using any of the registration techniquesdiscussed above with reference to FIGS. 20A-31. As discussed above, theregistration may produce a transformation matrix between the virtualtibia object with the patient's actual tibia bone. As discussed above,MR system 212 may display an animation, video, or text to describe how aparticular step or steps are to be performed. For instance, MR system212 may cause visualization device 213 to display a diagram or animationshowing how the registration process is to be performed. As alsodiscussed above, MR system 212 may display a virtual checklist, witheach item on the checklist items corresponding to an item in a checklistof steps of an orthopedic surgery. For instance, MR system 212 maydisplay a virtual checklist having a checklist item specifying a currentstep of registering a virtual tibia object with the patient's actualtibia bone.

The surgeon may perform various work steps to prepare the tibia bone(15006). Example work steps to prepare the tibia bone include, but arenot limited to, installing one or more guide pins into the tibia bone,drilling one or more holes in the tibia bone, and/or attaching one ormore guides to the tibia bone. MR system 212 may provide virtualguidance to assist the surgeon with the various work steps to preparethe tibia bone. As discussed above, MR system 212 may display ananimation, video, or text to describe how a particular step or steps areto be performed. For instance, MR system 212 may cause visualizationdevice 213 to display a diagram or animation showing how the tibia is tobe prepared. As also discussed above, MR system 212 may display avirtual checklist, with each item on the checklist items correspondingto an item in a checklist of steps of an orthopedic surgery. Forinstance, MR system 212 may display a virtual checklist having achecklist item specifying a current step, or sequence of steps, ofpreparing the tibia bone.

FIGS. 151A and 151B are conceptual diagrams illustrating exampleattachment of guide pins to a tibia. The incision process may expose atleast a portion of tibia 15102, fibula 15110, and talus 15108 of ankle15100. After performing the incision process, the surgeon may installguide pins 15104A, 15104B, 15106A, and 15106B into tibia 15102.

In some examples, such as the example of FIG. 151B, the surgeon mayinstall guide pins 15104A, 15104B, 15106A, and 15106B using a physicalguide. For instance, the surgeon may place tibial guide 15112 on tibia15102 and utilize one or more holes in tibial guide 15112 to guideinstallation of guide pins 15104A, 15104B, 15106A, and 15106B. In someexamples, tibial guide 15112 may be a patient-specific guide that ismanufactured with a surface designed to conform with the contours oftibia 15102. One example of such a patient specific guide is theProphecy Tibial Alignment Guide of the Prophecy® Infinity® Total Anklesystem produced by Wright Medical Group N.V.

In addition to, or in place of tibial guide 15112, MR system 212 mayprovide virtual guidance to assist the surgeon with the installation ofguide pins 15104A, 15104B, 15106A, and 15106B. For instance,visualization device 213 may display a virtual marker that guides asurgeon in installing a guide pin. Visualization device 213 may displaythe marker with an appearance that it is overlaid on tibia 15102 (e.g.,to indicate the position and/or orientation at which the guide pin is tobe installed). The virtual marker may be a virtual axis (e.g., similarto axis 3400 of FIG. 34) at a point on tibia 15102 that guides a surgeonin installing a guide pin. For instance, as shown in FIG. 151A,visualization device 213 may display virtual axes 15114A, 15114B,15116A, and 15116B to respectively guide installation of guide pins15104A, 15104B, 15106A, and 15106B, e.g., along the axes. While virtualaxes 15114A, 15114B, 15116A, and 15116B are illustrated in FIG. 151A asbeing displayed with an appearance similar to guide pins 15104A, 15104B,15106A, and 15106B of FIG. 151B, the display of virtual markers thatguide installation of guide pins (e.g., guide pins 15104A, 15104B,15106A, and 15106B) is not so limited. Other examples of virtual markersthat MR system 212 may display include, but are not limited to axes,arrows, points, circles, rings, polygons, X shapes, crosses, targets, orany other shape or combination of shapes. MR system 212 may display thevirtual markers as static features or with various animations or othereffects.

MR system 212 may utilize different types of virtual markers dependingon whether or not a physical guide is also used. As one example, in theexample of FIG. 151B where tibial guide 15112 is used, MR system 212 mayutilize an arrow to guide installation of a guide pin is to beinstalled. As shown in FIG. 151B, visualization device 213 may displayan arrow to guide installation of guide pin 15106A via a particular holeof tibial guide 15112. As another example, in the example of FIG. 151Awhere tibial guide 15112 is not used, MR system 212 may utilize avirtual axis to guide installation of a guide pin. As shown in FIG.151A, visualization device 213 may display virtual axis 15116A to guideinstallation of guide pin 15106A.

In examples where multiple guide pins are to be installed, visualizationdevice 213 may display a respective virtual marker for each guide pin.In the example of FIG. 151, visualization device 213 may displaymultiple virtual markers to guide installation of guide pins 15104A,15104B, 15106A, and 15106B. In some examples, visualization device 213may display the virtual markers concurrently. For instance,visualization device 213 may display virtual axes 15114A, 15114B,15116A, and 15116B, e.g., for alignment of guide pins, at the same time.In other examples, visualization device 213 may display fewer than allof the virtual markers at a particular time. For instance, visualizationdevice 213 may display the virtual markers sequentially. As one example,at a first time, visualization device 213 may display a first virtualmarker that guides installation of a first guide pin (e.g., guide pin15104A). At a second time that is after the first time (e.g., afterguide pin 15104A has been installed), visualization device 213 maydisplay a second virtual marker that guides installation of a secondguide pin (e.g., guide pin 15104B). In other words, responsive todetermining that guide pin 15404A has been installed, visualizationdevice 213 may cease to display the virtual marker that guidedinstallation of guide pin 15404A and display a virtual marker to a nextguide pin to be installed. Visualization device 213 may continue tosequentially display virtual markers until all necessary guide pins areinstalled (e.g., until guide pins 15104A, 15104B, 15106A, and 15106B areinstalled). In this way, MR system 212 may display a plurality of avirtual axes each having parameters obtained from the virtual surgicalplan, each of the virtual axes configured to guide installation of arespective guide pin of a plurality of pins in the tibia.

MR system 212 may display the virtual markers with particular colors.For instance, in some examples, MR system 212 may preferably display thevirtual markers in a color other than red, such as green, blue, yellow,etc. Displaying the virtual markers in a color or colors other than redmay provide one or more benefits. For instance, as blood appears red andblood may be present on or around the anatomy of interest, a red coloredvirtual marker may not be visible.

In some examples, such as where visualization system 213 displaysmultiple virtual markers at the same time, visualization system 213 mayalter or otherwise modify the display of a virtual marker after thesurgeon has completed a corresponding work step. Alterations of thedisplay of virtual markers may include, but are not limited to, changinga color, changing a marker type, animating (e.g., blinking or flashing),displaying an additional element (e.g., an X or a checkmark on or nearthe virtual marker) or any other visually perceptible alteration. Forinstance, visualization system 213 may initially display a first virtualmarker to guide installation of guide pin 15104A as a virtual axis and asecond virtual marker to guide installation of guide pin 15104B as avirtual axis. After the surgeon installs guide pin 15104A, visualizationsystem 213 may modify the first virtual marker displayed to guideinstallation of guide pin 15104A (e.g., changing from a virtual axis toa reticle) while maintaining the display of the second virtual marker asa virtual axis.

MR system 212 may provide other virtual guidance in addition to, or inplace of, the virtual markers. For instance, MR system 212 may displaydepth guidance to enable the surgeon to install the guide pins to atarget depth (e.g., depth guidance similar to the depth guidancediscussed above with reference to FIGS. 66-68). As another example, MRsystem 212 may provide targeting guidance. For instance, MR system 212may display one or both of a virtual marker that identifies a centerpoint or prescribed axis of the pin installation (e.g., as discussedabove with reference to FIGS. 36A-36D) and/or an indication of whetherthe guide pin is aligned with the prescribed axis. As discussed above,MR system 212 may determine whether the guide pin is aligned with theprescribed axis by monitoring a position/orientation of the guide pinand/or a drill driving the guide pin, and comparing the monitoredposition/orientation with the prescribed axis.

The surgeon may install guide pins 15104A, 15104B, 15106A, and 15106Busing the virtual guidance. In examples where tibial guide 15112 wasused, the surgeon may remove tibial guide 15112 after installation ofguide pins 15104A, 15104B, 15106A, and 15106B.

FIG. 152 is a conceptual diagram illustrating example drilling of holesin a tibia. As shown in FIG. 152, the surgeon may install drilling guide15202 onto tibia 15102 using guide pins 15104A, 15104B, 15106A, and15106B. Drilling guide 15202 includes one or more channels that guidedrilling of holes into tibia 15102. For instance, as shown in FIG. 152,drilling guide 15202 include first channel 15204A and second channel15204B. The surgeon may utilize a drill (e.g., a surgical motor withtibial corner drill bit) to drill a hole using each of first channel15204A and second channel 15204B. In this way, the surgeon maybi-cortically drill both proximal corners of tibia 15102.

In addition to, or in place of drilling guide 15202, MR system 212 mayprovide virtual guidance to assist the surgeon with the drilling of theproximal corners of tibia 15102. For instance, visualization device 213may display a virtual marker that guides a surgeon in drilling a hole intibia 15102. Visualization device 213 may display the virtual markeroverlaid on tibia 15102 (e.g., to indicate the position and/ororientation at which the hole is to be drilled). The virtual marker maybe a virtual drilling axis (e.g., similar to axis 3400 of FIG. 34) at apoint on tibia 15102 that guides a surgeon in performing the drilling.Similar to the virtual markers discussed above that guide installationof guide pins, visualization device 213 device may display the virtualmarkers that guide the drilling of the proximal corners of tibia 15102concurrently or sequentially, and the virtual markers that guide thedrilling may each respective proximal corner of the tibia.

MR system 212 may provide other virtual guidance in addition to, or inplace of, the virtual markers. For instance, MR system 212 may displaydepth guidance to enable the surgeon to drill the holes to a targetdepth (e.g., depth guidance similar to the depth guidance discussedabove with reference to FIGS. 66-68). As another example, MR system 212may provide targeting guidance. For instance, MR system 212 may displayone or both of a virtual marker that identifies a center point orprescribed axis of the drilling (e.g., as discussed above with referenceto FIGS. 36A-36D), e.g., into the tibia or talus, and/or an indicationof whether the drill bit is aligned with the prescribed axis. Asdiscussed above, MR system 212 may determine whether the drill bit isaligned with the prescribed axis by monitoring a position/orientation ofthe drill bit and/or a drill driving the drill bit, and comparing themonitored position/orientation with the prescribed axis.

In some examples, MR system 212 may utilize the surgical item trackingtechniques described in this disclosure (e.g., with reference to FIGS.83-108) to assist with the ankle arthroplasty. For instance, MR system212 may select a surgical item of a plurality of surgical items. Theplurality of surgical items may be included in one or more trays forperforming the ankle arthroplasty procedure. Example surgical itemsinclude, but are not limited to, tools, instruments, implants,associated hardware. MR system 212 may cause a second visualizationdevice (e.g., worn by a nurse or someone other than the surgeon) todisplay virtual information that identifies the selected surgical itemamong the plurality of surgical items, wherein the virtual informationis presented on or adjacent a position of the selected surgical itemvisible via the second visualization device. MR system 212 may selectthe surgical item as a surgical item associated with a current step ofthe ankle arthroplasty procedure. For instance, in the example of FIG.152 where the surgeon may use drilling guide 15202 to drill the tibialcorners, MR system 212 may select drilling guide 15202 as the selectedsurgical item. The second visualization device may display virtualinformation that identifies drilling guide 15202 (e.g., highlight orotherwise identify drilling guide 15202 in a manner similar to FIGS.105-108)

With continued reference to the stages of an ankle joint repair surgeryof FIG. 150, the surgeon may perform a tibia resection process (15008).For instance, the surgeon may remove a portion of tibia 15102 to makeroom for subsequent installation of a tibial implant. In some examples,the surgeon may perform the tibial resection by making three cuts (e.g.,a proximal cut, a medial cut, and a lateral cut) in tibia 15102 toremove a portion of tibia 15102 and create a space for subsequentinstallation of a tibial implant. As discussed above, MR system 212 maydisplay an animation, video, or text to describe how a particular stepor steps are to be performed. For instance, MR system 212 may causevisualization device 213 to display a diagram or animation showing howthe tibia resection is to be performed. As also discussed above, MRsystem 212 may display a virtual checklist, with each item on thechecklist items corresponding to an item in a checklist of steps of anorthopedic surgery. For instance, MR system 212 may display a virtualchecklist having a checklist item specifying a current step, or sequenceof steps, of performing the tibial resection.

FIG. 153 is a conceptual diagram illustrating example resection of atibia. As shown in FIG. 153, the surgeon may install resection guide15302 onto tibia 15102 using guide pins 15104A, 15104B, 15106A, and15106B. Resection guide 15302 includes one or more channels that guideperforming cuts into tibia 15102. For instance, as shown in FIG. 153,resection guide 15202 include first channel 15306A that guidesperformance of a medial cut, second channel 15306B that guidesperformance of a proximal cut, and third channel 15306C that guidesperformance of a lateral cut. In some examples, resection guide 15302may include a fourth channel that guides performance of a resection oftalus 15108. For instance, as shown in FIG. 153, resection guide 15302may include fourth channel 15304. The surgeon may utilize a saw blade(e.g., an oscillating bone saw) to perform the medial, lateral, andproximal cuts using channels 15306A-15306C. In this way, the surgeon mayperform a resection of tibia 15102.

In addition to, or in place of resection guide 15302, MR system 212 mayprovide virtual guidance to assist the surgeon with performing theresection of tibia 15102. For instance, visualization device 213 maydisplay a virtual marker that guides a surgeon in performing a cut intibia 15102. Visualization device 213 may display the marker overlaid ontibia 15102 (e.g., to indicate the position and/or orientation at whichthe cut is to be made). The virtual marker may be a virtual cuttingline, a virtual cutting surface or a virtual cutting plane (e.g.,similar to virtual cutting plane 4200 of FIGS. 42A and 42B) at a pointon tibia 15102 that guides a surgeon in performing the cut. Similar tothe virtual markers discussed above that guide installation of guidepins, visualization device 213 device may display the virtual markersthat guide the performance of the proximal, medial, and lateral cutsconcurrently or sequentially. In this way, MR system 212 may display aplurality of virtual cutting surfaces each having parameters obtainedfrom the virtual surgical plan, the plurality of virtual cuttingsurfaces configured to guide resection of the tibia.

MR system 212 may provide other virtual guidance in addition to, or inplace of, the virtual markers. For instance, MR system 212 may displaydepth guidance to enable the surgeon to perform the cuts to a targetdepth (e.g., depth guidance similar to the depth guidance discussedabove with reference to FIGS. 66-68). As another example, MR system 212may provide targeting guidance. For instance, MR system 212 may displayone or both of a virtual marker that identifies a prescribed plane ofthe cutting (e.g., as discussed above with reference to FIGS. 36A-36D)and/or an indication of whether the saw blade is aligned with theprescribed plane. As discussed above, MR system 212 may determinewhether the saw blade is aligned with the prescribed plane by monitoringa position/orientation of the saw blade and/or a motor driving the sawblade the guide pin, and comparing the monitored position/orientationwith the prescribed plane.

The surgeon may remove the resection (i.e., the portion of tibia 15102separated via the cuts). Guide pins 15104A and 15104B may be attached tothe resection and removed as a consequence of the resection removal.

As discussed above, MR system 212 may cause the second visualizationdevice to display virtual information that identifies surgical itemselected for a current step of the ankle arthroplasty procedure. Forinstance, in the example of FIG. 153 where the surgeon may use resectionguide 15302 to perform the tibial resection, MR system 212 may selectresection guide 15302 as the selected surgical item.

Furthermore, with reference to the stages of the ankle joint repairsurgery of FIG. 150, the surgeon may perform a registration process thatregisters a virtual talus object with the patient's actual talus bone(15010) in the field of view presented to the surgeon by visualizationdevice 213. For instance, MR system 212 may obtain the virtual talusobject from storage system 206 of FIG. 2. Similar to the virtual tibiaobject discussed above, the virtual talus object may be generated basedon pre-operative imaging (e.g., CT imaging) of the patient's talus. MRsystem 212 may perform the registration using any suitable process. Forinstance, MR system 212 may perform the registration of the virtualtalus object with the patient's actual talus bone using any of theregistration techniques discussed above with reference to FIGS. 20A-31.As discussed above, the registration may produce a transformation matrixbetween the virtual talus object with the patient's actual talus bone.As also discussed above, MR system 212 may display a virtual checklist,with each item on the checklist items corresponding to an item in achecklist of steps of an orthopedic surgery. For instance, MR system 212may display a virtual checklist having a checklist item specifying acurrent step, or sequence of steps, of registering a virtual talusobject with the patient's actual talus bone.

Additionally, in the example of FIG. 150, the surgeon may performvarious work steps to prepare the talus bone (15012). Example work stepsto prepare the talus bone include, but are not necessarily limited to,installing one or more guide pins into the talus bone, drilling one ormore holes in the talus bone, and/or attaching one or more guides (e.g.,cutting guides, drilling guides, reaming guides, etc.) to the talusbone. MR system 212 may provide virtual guidance to assist the surgeonwith the various work steps to prepare the talus bone. As discussedabove, MR system 212 may display an animation, video, or text todescribe how a particular step or steps are to be performed. Forinstance, MR system 212 may cause visualization device 213 to display adiagram or animation showing how the talus is to be prepared. As alsodiscussed above, MR system 212 may display a virtual checklist, witheach item on the checklist items corresponding to an item in a checklistof steps of an orthopedic surgery. For instance, MR system 212 maydisplay a virtual checklist having a checklist item specifying a currentstep, or sequence of steps, of preparing the talus bone.

FIGS. 154A and 154B are conceptual diagrams illustrating example guidepins installed in a talus during the talus preparation process. As shownin FIGS. 154A and 154B, the surgeon may install guide pins 15402A and15402B into talus 15108.

In some examples, such as the example of FIG. 154B, the surgeon mayinstall guide pins 15402A and 15402B using a physical guide. Forinstance, the surgeon may place talar guide 15404 on talus 15108 andutilize one or more holes in talar guide 15404 to guide installation ofguide pins 15402A and 15402B. In some examples, talar guide 1540 may bea patient-specific guide that is manufactured with a surface designed toconform with the contours of talus 15108. One example of such apatient-specific guide is the Prophecy Talus Alignment Guide of theProphecy® Infinity® Total Ankle system produced by Wright Medical GroupN.V.

In addition to, or in place of talar guide 15404, MR system 212 mayprovide virtual guidance to assist the surgeon with the installation ofguide pins 15402A and 15402B. For instance, visualization device 213 maydisplay one or more virtual markers that guide a surgeon in installing aguide pin of guide pins 15402A and 15402B. For instance, as shown inFIG. 154A, visualization device 213 may display virtual axes 15406A and15406B to respectively guide installation of guide pins 15402A and15402B. Visualization device 213 may display the virtual markers in amanner similar to that described above with reference to FIGS. 151A and151B. MR system 212 may provide other virtual guidance to assist withthe installation of guide pins 15402A and 15402B in addition to, or inplace of, the virtual markers. For instance, MR system 212 may provideany of the additional virtual guidance (e.g., depth guidance, targetingguidance, etc.) discussed above. In this way, MR system 212 may displaya plurality of a virtual axes each having parameters obtained from thevirtual surgical plan, and each of the virtual axes configured to guideinstallation of a respective guide pin in the talus. A virtual axis mayguide installation of a corresponding guide pin by providing a visualreference with which a surgeon may align the physical guide pin duringinstallation of the guide pin. As discussed herein, in some examples, MRsystem 212 may provide feedback as to whether the physical guide pin isactually aligned with the virtual axis.

The surgeon may install guide pins 15402A and 15402B using the virtualguidance. For example, the surgeon may align guide longitudinal axes ofpins 15402A and 15402B with respective virtual axes to place the pins inbone. In examples where talar guide 15404 was used, the surgeon mayremove talar guide 15404 after installation of guide pins 15402A and15402B.

As discussed above, MR system 212 may cause the second visualizationdevice to display virtual information that identifies a surgical itemselected for a current step of the ankle arthroplasty procedure. Forinstance, in the example of FIG. 154B where the surgeon may use talarguide 15404 to install guide pins 15402A and 15402B, MR system 212 mayselect talar guide 15404 as the selected surgical item.

With continued reference to FIG. 150, after performing the taluspreparation process, the surgeon may perform various perform a talusresection process (15014). For instance, the surgeon may remove aportion of talus 15108 to make room for subsequent installation of atalus implant. In some examples, the surgeon may perform the talusresection by making a single cut in talus 15108 to remove a portion oftalus 15108 and create a space for subsequent installation of a talusimplant. As discussed above, MR system 212 may display an animation,video, or text to describe how a particular step or steps are to beperformed. For instance, MR system 212 may cause visualization device213 to display a diagram or animation showing how the talus resection isto be performed. As also discussed above, MR system 212 may display avirtual checklist, with each item on the checklist items correspondingto an item in a checklist of steps of an orthopedic surgery. Forinstance, MR system 212 may display a virtual checklist having achecklist item specifying a current step, or sequence of steps, ofperforming the talar resection.

FIG. 155 is a conceptual diagram illustrating example resection of atalus. As shown in FIG. 155, the surgeon may install resection guide15302 onto talus 15108 using guide pins 15402A and 15402B. In theexample of FIG. 155, the surgeon may utilize the same resection guide(i.e., resection guide 15302) as was used to perform the tibialresection. In other examples, a talus specific resection guide may beused. The surgeon may perform the talus resection using resection guide15302. For instance, the surgeon may utilize a saw blade (e.g., anoscillating bone saw) to perform a cut using channel 15304. In this way,the surgeon may perform a resection of talus 15108.

In addition to, or in place of resection guide 15302, MR system 212 mayprovide virtual guidance to assist the surgeon with performing theresection of talus 15308. For instance, visualization device 213 maydisplay a virtual marker that guides a surgeon in performing a cut intalus 15108. Visualization device 213 may display the marker overlaid ontalus 15108 (e.g., to indicate the position and/or orientation at whichthe cut is to be made). The virtual marker may be a virtual cuttingline, virtual cutting surface or virtual cutting plane (e.g., similar tovirtual cutting plane 4200 of FIGS. 42A and 42B) at a point on talus15108 that guides a surgeon in performing the cut. In this way, MRsystem 212 may display a virtual cutting surface having parametersobtained from the virtual surgical plan, the virtual cutting surfaceconfigured to guide primary resection of the talus.

MR system 212 may provide other virtual guidance in addition to, or inplace of, the virtual markers. For instance, MR system 212 may displaydepth guidance to enable the surgeon to perform the cut to a targetdepth (e.g., depth guidance similar to the depth guidance discussedabove with reference to FIGS. 66-68). As another example, MR system 212may provide targeting guidance. For instance, MR system 212 may displayone or both of a virtual marker that identifies a prescribed plane ofthe cutting (e.g., as discussed above with reference to FIGS. 36A-36D)and/or an indication of whether the saw blade is aligned with theprescribed plane. As discussed above, in some examples, MR system 212may determine whether the saw blade is aligned with the prescribed planeby registering the saw blade or something connected thereto (e.g., a sawmotor body, a saw handle, a physical registration marker, etc.) with acorresponding virtual model, and comparing the position of thecorresponding virtual model with the prescribed plane.

The surgeon may remove the resection (i.e., the portion of talus 15108separated via the cuts). In some examples, the surgeon may use varioustools (e.g., a reciprocating saw or bone rasp) to remove any excess boneleft after the resection has been removed. FIG. 156 is a conceptualdiagram of an example ankle after performance of a tibial resection anda talar resection.

The surgeon may perform one or more additional work steps on one or bothof tibia 15102 and/or talus 15108 to prepare tibia 15102 and/or talus15108 to receive implants. Example additional work steps include, butare not necessarily limited to, tibial tray trialing, tibial pegbroaching, talar chamfer resections, and talar peg drilling.

FIGS. 157A-157C are conceptual diagrams illustrating an example oftibial tray trialing. In some examples, it may be desirable to ensurethat, when installed, a posterior edge of the tibial implant will atleast reach the posterior portion of tibia 15102. Additionally, in someexamples, there may be multiple size tibial implants available. As such,it may be desirable for the surgeon to determine which size tibialimplant to utilize. To ensure that the posterior edge of the tibialimplant will at least reach the posterior portion of tibia 15102 and/orto determine which size tibial implant to utilize, the surgeon mayperform tibial tray trialing.

To perform tibial tray trialing, the surgeon may attach tibial traytrial 15702 to tibia 15102. As shown in FIG. 157A, tibial tray trial15702 may include posterior edge 15704, indicator 15710, guide pin holes15712A and 15712B, broaching holes 15714A and 15714B (an additionalanterior broaching hole 15714C is not shown), and anterior surface15716. The surgeon may attach tibial tray trial 15702 to tibia 15102 bysliding guide pins 15106A and 15106B into corresponding guide pin holes15712A and 15712B. In some examples, after attaching tibial tray trial15702, the surgeon may trim guide pins 15106A and 15106B to be flushwith anterior surface 15716 of tibial tray trial 15702 (e.g., as shownin FIG. 158).

In some examples, the surgeon may utilize fluoroscopy to perform thetibial tray trialing. For instance, the surgeon may utilize fluoroscopyto determine the relative positions of tibial tray trial 15702 and tibia15102.

MR system 212 may provide virtual guidance to assist with tibial traytrialing. As one example, visualization device 213 may display asynthesized view showing the relative positions of tibial tray trial15702 and tibia 15102. Visualization device 213 may display thesynthesized view in a manner similar to that discussed above withreference to FIGS. 66-68. For instance, MR system 212 may registertibial tray trial 15702 to a corresponding virtual model of tibial traytrial and utilize the registered virtual models of tibial tray trial15702 and tibia 15102 to synthesize a view showing the relativepositions of the virtual models of tibial tray trial 15702 and tibia15102. As the virtual models of tibial tray trial 15702 and tibia 15102are respectively registered to tibial tray trial 15702 and tibia 15102,the relative positions of the virtual models of tibial tray trial 15702and tibia 15102 corresponds to the relative positions of tibial traytrial 15702 and tibia 15102. The synthesized views may appear similar tothe conceptual diagrams of FIGS. 157B and 157C.

The surgeon may utilize the synthesized view to perform one or moreadjustments on tibial tray trial 15702. For instance, if the synthesizedview indicates that posterior edge 15704 of tibial tray trial 15702extends past posterior edge 15706 of tibia 15102, the surgeon may adjusttibial tray trial 15702 to anteriorly advance posterior edge 15704 oftibial tray trial 15702. For instance, the surgeon may utilize tool15708 to anteriorly translate tibial tray trial 15702.

The surgeon may utilize the synthesized view to determine which sizetibial implant is to be utilized. For instance, if the synthesized viewindicates that indicator 15710 (illustrated in FIG. 157C as a notch) oftibial tray trial 15702 extends past posterior edge 15706 of tibia15102, the surgeon may determine that a first size tibial implant (e.g.,a standard size) is to be utilized. If the synthesized view indicatesthat indicator 15710 of tibial tray trial 15702 does not extend pastposterior edge 15706 of tibia 15102, the surgeon may determine that asecond size tibial implant (e.g., a long size) is to be utilized.

As described above, MR system 212 may enable the surgeon to performtibial tray trialing using virtual guidance. In some examples, MR system212 may enable the surgeon to perform tibial tray trialing without usingfluoroscopy.

As discussed above, MR system 212 may cause the second visualizationdevice to display virtual information that identifies surgical itemselected for a current step of the ankle arthroplasty procedure. Forinstance, in the example of FIGS. 157A-157C where the surgeon may usetibial tray trial 15702, MR system 212 may select tibial tray trial15702 as the selected surgical item.

The surgeon may create anchorage points for the tibial implant. Forinstance, the surgeon may utilize tibial tray trial to perform tibialpeg broaching. FIG. 158 is a conceptual diagram illustrating an examplecreation of tibial implant anchorage. As shown in FIG. 158, the surgeonmay utilize anterior tibial peg broach 15802A to broach a first anteriorhole in tibia 15102 using broaching hole 15714A, utilize anterior tibialpeg broach 15802A to broach a second anterior hole in tibia 15102 usingbroaching hole 15714C, and utilize posterior tibial peg broach 15802B tobroach a hole in tibia 15102 using broaching hole 15714B. The holesbroached in tibia 15102 may constitute anchorage points for the tibialimplant.

As discussed above, MR system 212 may cause the second visualizationdevice to display virtual information that identifies surgical itemselected for a current step of the ankle arthroplasty procedure. Forinstance, in the example of FIG. 158 where the surgeon may use anteriortibial peg broach 15802A and posterior tibial peg broach 15802B, MRsystem 212 may select anterior tibial peg broach 15802A and posteriortibial peg broach 15802B as the selected surgical item (or items). Asdiscussed above, MR system 212 may cause the second visualizationdevice, and/or visualization device 213, to visually distinguish theselected surgical items (i.e., anterior tibial peg broach 15802A andposterior tibial peg broach 15802B).

The surgeon may perform one or more talar chamfer resections to furtherprepare talus 15108 to receive the talar implant. In some examples, thesurgeon may perform an anterior talar chamfer resection and a posteriortalar chamfer resection. To perform the one or more talar resections,the surgeon may attach one or more guide pins to talus 15108.

FIGS. 159A and 159B are conceptual diagrams illustrating an exampleattachment of guide pins to talus 15108. MR system 212 may providevirtual guidance to guide the surgeon in attaching guide pins 15904A and15904B to talus 15108. For instance, as shown in FIG. 159A,visualization device 213 may display virtual axes 15902A and 15902Boverlaid on talus 15108 to guide installation of guide pins 15904A and15904B to talus 15108. While illustrated in FIG. 159A as virtual axes,visualization device 213 may display any of the virtual markersdescribed herein to guide installation of guide pins 15904A and 15904Bto talus 15108.

In some examples, the surgeon may utilize a physical guide to assistwith the installation of guide pins 15904A and 15904B to talus 15108.For instance, the surgeon may utilize fluoroscopy to position a talardome trial component. When the talar dome trial component is positioned,the surgeon may utilize holes in the talar dome trial component to guidethe installation of guide pins 15904A and 15904B.

The surgeon may perform the talar chamfer resections using guide pins15904A and 15904B. For instance, as shown in FIG. 160, the surgeon mayposition talar resection guide base 16002 on talus 15108 using guidepins 15904A and 15904B. The surgeon may utilize one or more componentsto secure talar resection guide base 16002 to talus 15108. For instance,as shown in FIG. 161, the surgeon may install fixation screws 16102A and16102B through resection guide base 16002 into talus 15108.

MR system 212 may provide virtual guidance to assist the surgeon withthe installation of fixation screws 16102A and 16102B. As one example,visualization device 213 may display virtual markers that indicate thelocation and axis at which fixation screws 16102A and 16102B are to beinstalled. As another example, visualization device 213 may providedepth guidance to enable the surgeon to install fixation screws 16102Aand 16102B to a target depth (e.g., depth guidance similar to the depthguidance discussed above with reference to FIGS. 66-68). In someexamples, MR system 212 may utilize closed-loop tool control topositively control a drill used to attach fixation screws 16102A and16102B. For instance, MR system 212 may utilize the closed-loop toolcontrol techniques discussed above, e.g., with reference to FIG. 72, toreduce a speed of and/or stop the drill used to attach fixation screws16102A and 16102B when a desired depth and/or torque is reached.

The surgeon may utilize talar resection guide base 16002 to perform theposterior talar chamfer resection. For instance, as shown in FIG. 161,the surgeon may insert saw blade 16104 into slot 16004 of talarresection guide base 16002 to perform the posterior talar chamferresection.

As discussed above, MR system 212 may cause the second visualizationdevice to display virtual information that identifies surgical itemselected for a current step of the ankle arthroplasty procedure. Forinstance, in the example of FIG. 161 where the surgeon may use talarresection guide base 16002, MR system 212 may select talar resectionguide base 16002 as the selected surgical item.

In addition to, or in place of talar resection guide base 16002, MRsystem 212 may provide virtual guidance to assist the surgeon withperforming the posterior talar chamfer resection. For instance,visualization device 213 may display a virtual marker that guides asurgeon in performing the posterior talar chamfer resection.Visualization device 213 may display the marker overlaid on talus 15108(e.g., to indicate the position and/or orientation at which the cut isto be made). The virtual marker may be a virtual surface or virtualcutting plane (e.g., similar to virtual cutting plane 4200 of FIGS. 42Aand 42B) at a point on talus 15108 that guides a surgeon in performingthe cut.

The surgeon may utilize talar resection guide base 16002 to perform theanterior talar chamfer resection. For instance, as shown in FIG. 162,the surgeon may attach anterior talar guide 16202 to talar resectionguide base 16002. The surgeon may utilize a drill with talar reamer16204 to ream the anterior surface of talus 15108. For instance, thesurgeon may slide talar reamer 16204 horizontally through anterior talarguide 16202 to prepare the surface of talus 15108 for an anterior flatof the talar implant. As shown in FIG. 162, talar reamer 16204 mayinclude depth stop 16206 that engages surface 16208 of anterior talarguide 16202 to control the reaming depth. The surgeon may rotate talarguide 16202 180 degrees and again slide talar reamer 16204 horizontallythrough (the now rotated) anterior talar guide 16202 to prepare thesurface of talus 15108 for an anterior chamfer of the talar implant. Asdiscussed above, talar reamer 16204 may include depth stop 16206 thatengages surface 16208 of anterior talar guide 16202 to control thereaming depth.

In some examples, for one or both of the anterior flat and anteriorchamfer preparation, the surgeon may perform plunge cuts (e.g., usingtalar reamer 16204) to prepare talus 15108 for reaming. For instance,the surgeon may attach a pilot guide with holes that guide performanceof the plunge cuts. Depth stop 16206 of talar reamer 16204 may engagewith a surface of the pilot guide the control the plunge depth.

In addition to, or in place of talar resection guide base 16002, MRsystem 212 may provide virtual guidance to assist the surgeon withperforming the anterior talar chamfer resection. For instance,visualization device 213 may display one or more virtual markers thatguide a surgeon in performing the plunge cuts and/or horizontal reaming.As one example, visualization device 213 may display a respectivevirtual axis for each of the plunge cuts. MR system 212 may provideother virtual guidance to assist with performing the plunge cuts and/orhorizontal reaming in addition to, or in place of, the virtual markers.For instance, MR system 212 may provide any of the additional virtualguidance (e.g., depth guidance, targeting guidance, etc.) discussedabove.

The surgeon may perform talar peg drilling to create anchorage points intalus 15108 for the talar implant. MR system 212 may provide virtualguidance to assist the surgeon with performing the anterior talarchamfer resection. For instance, visualization device 213 may displayone or more virtual markers that guide a surgeon in drilling holes intalus 15108. As shown in FIG. 164, visualization device 213 may displayvirtual axes 16402A and 16402B that guide drilling of peg holes 16502Aand 16502B of FIG. 165. MR system 212 may provide other virtual guidanceto assist with creating the anchorage in addition to, or in place of,the virtual markers. For instance, MR system 212 may provide any of theadditional virtual guidance (e.g., depth guidance, targeting guidance,etc.) discussed above. In this way, MR system 212 may display aplurality of virtual drilling axes each having parameters obtained fromthe virtual surgical plan, each of the virtual drilling axes configuredto guide drilling of an anchorage point in the talus.

With continued reference to FIG. 150, the surgeon may perform a tibiaimplant installation process (15016). FIG. 166 is a conceptual diagramillustrating an example tibial implant. As shown in FIG. 166, tibialimplant 16602 includes posterior peg 16604A, and anterior pegs 16604Band 16604C. FIG. 167 is a conceptual diagram illustrating an exampletibia as prepared using the steps described above. As shown in FIG. 167,tibia 15102 includes peg holes 16702A-16702C that were created duringthe broaching process described above with reference to FIG. 158.

The surgeon may install tibial implant 16602 such that posterior peg16604A, and anterior pegs 16604B and 16604C of tibial implant 16602engage with peg holes 16702A-16702C of tibia 15102. For instance, thesurgeon may position tibial implant 16602 such that posterior peg 16604Alines up with peg hole 16702A, anterior peg 16604B lines up with peghole 16702B, and anterior peg 16604C lines up with peg hole 16702C. Oncethe pegs are lined up with their corresponding peg holes, the surgeonmay impact tibial implant 16602 into tibia 15102. As discussed above, MRsystem 212 may display an animation, video, or text to describe how aparticular step or steps are to be performed. For instance, MR system212 may cause visualization device 213 to display a diagram or animationshowing how tibial implant 16602 is to be installed. As also discussedabove, MR system 212 may display a virtual checklist, with each item onthe checklist items corresponding to an item in a checklist of steps ofan orthopedic surgery. For instance, MR system 212 may display a virtualchecklist having a checklist item specifying a current step, or sequenceof steps, of installing the tibial implant.

FIG. 168 is a conceptual diagram illustrating example impaction of atibial implant into a tibia. As shown in FIG. 168, the surgeon mayutilize tray impactor 16802 to impact tibial implant 16602 into tibia15102. For instance, the surgeon may place tip 16806 of tray impactor16802 on tibial implant 16602 and strike one or both of impaction points16804A and/or 16804B with an impactor (e.g., a hammer).

With continued reference to FIG. 150, the surgeon may perform a talusimplant installation process (15018). FIG. 169 is a conceptual diagramillustrating an example talar implant. As shown in FIG. 169, talarimplant 16902 includes first peg 16904A and second peg 16904B.

The surgeon may install talar implant 16902 such that first peg 16904Aand second peg 16904B of talar implant 16902 engage with peg holes16502A and 16502B of talus 15108. For instance, the surgeon may positiontalar implant 16902 such that first peg 16904A lines up with peg hole16502A, and second peg 16904B of talar implant 16902 lines up with peghole 16502B. Once the pegs are lined up with their corresponding pegholes, the surgeon may impact talar implant 16902 into talus 15108.

As discussed above, MR system 212 may cause the second visualizationdevice to display virtual information that identifies a surgical itemselected for a current step of the ankle arthroplasty procedure. Forinstance, in the example of FIG. 168 where the surgeon may use trayimpactor 16802, MR system 212 may select tray impactor 16802 as theselected surgical item.

FIG. 170 is a conceptual diagram illustrating example impaction of atalar implant into a talus. As shown in FIG. 170, the surgeon mayutilize talar impactor 17002 to impact talar implant 16902 into talus15108. For instance, the surgeon may place tip 17004 of talar impactor17002 on talar implant 16902 and strike an impaction point of talarimpactor 17002 with an impactor (e.g., a hammer).

As discussed above, MR system 212 may cause the second visualizationdevice to display virtual information that identifies surgical itemselected for a current step of the ankle arthroplasty procedure. Forinstance, in the example of FIG. 168 where the surgeon may use talarimpactor 17002, MR system 212 may select talar impactor 17002 as theselected surgical item. As discussed above, MR system 212 may display ananimation, video, or text to describe how a particular step or steps areto be performed. For instance, MR system 212 may cause visualizationdevice 213 to display a diagram or animation showing how talar implant16902 is to be installed. As also discussed above, MR system 212 maydisplay a virtual checklist, with each item on the checklist itemscorresponding to an item in a checklist of steps of an orthopedicsurgery. For instance, MR system 212 may display a virtual checklisthaving a checklist item specifying a current step, or sequence of steps,of installing the talar implant.

With continued reference to FIG. 150, the surgeon may perform a bearinginstallation process (15020). The surgeon may install a bearing betweentibial implant 16602 and talar implant 16902. For instance, as shown inFIG. 171, the surgeon may install bearing 17102 between tibial implant16602 and talar implant 16902. As discussed above, MR system 212 maydisplay an animation, video, or text to describe how a particular stepor steps are to be performed. For instance, MR system 212 may causevisualization device 213 to display a diagram or animation showing howbearing 17102 is to be installed. As also discussed above, MR system 212may display a virtual checklist, with each item on the checklist itemscorresponding to an item in a checklist of steps of an orthopedicsurgery. For instance, MR system 212 may display a virtual checklisthaving a checklist item specifying a current step, or sequence of steps,of installing the bearing.

Subsequently, in the example of FIG. 150, the surgeon may perform awound closure process (15022). During the wound closure process, thesurgeon may reconnect tissues severed during the incision process inorder to close the wound in the patient's ankle. As discussed above, MRsystem 212 may display an animation, video, or text to describe how aparticular step or steps are to be performed. For instance, MR system212 may cause visualization device 213 to display a diagram or animationshowing how the wound is to be closed. As also discussed above, MRsystem 212 may display a virtual checklist, with each item on thechecklist items corresponding to an item in a checklist of steps of anorthopedic surgery. For instance, MR system 212 may display a virtualchecklist having a checklist item specifying a current step, or sequenceof steps, of closing the wound.

FIG. 172 is a flow diagram illustrating an example technique for MRaided surgery, in accordance with one or more techniques of thisdisclosure. As discussed above, a surgeon may wear a visualizationdevice of an MR system, such as visualization device 213 of MR system212.

MR system 212 may register, via a visualization device, a virtual modelof a portion of an anatomy of a patient to a corresponding portion ofthe anatomy viewable via the visualization device (17202). For instance,MR system 212 may perform the registration of a virtual tibia objectwith the patient's actual tibia bone viewable via visualization device213 using any of the registration techniques discussed above withreference to FIGS. 20A-31. As discussed above, MR system 212 may obtainthe virtual model from a virtual surgical plan for a surgical procedureto attach a prosthetic to the anatomy.

As discussed above, visualization device 213 may display at least aportion of the virtual model during the registration of the virtualmodel of the portion of the anatomy to the corresponding portion of theanatomy. For instance, as shown in FIG. 24, visualization device 213 maydisplay a virtual model (e.g., a virtual bone model) and the user (i.e.,the wearer of visualization device 213) may shift their gaze to alignthe virtual model with the corresponding portion of the anatomy. In someexamples, visualization device 213 may cease display of the virtualmodel after the registration process. In some examples, visualizationdevice 213 may maintain the display of the virtual model after theregistration process and/or re-display the virtual model at a later timeduring the surgical procedure.

MR system 212 may display, via the visualization device and overlaid onthe portion of the anatomy, a virtual guide that guides at least one ofpreparation of the anatomy for attachment of the prosthetic orattachment of the prosthetic to the anatomy (17204). For instance, MRsystem 212 may cause visualization device 213 to display any of thevirtual guidance/guides discussed above. As discussed above, examplevirtual guides include, but are not limited to, virtual axes, virtualcutting surfaces, and the like. As also discussed above, visualizationdevice 213 may display the virtual guide overlaid on the portion of theanatomy such that the virtual guide appears to be part of the real-worldscene, e.g., with the virtual guide appearing to the user to be inoverlay or otherwise integrated within the actual, real-world scene.

As discussed above, in some examples, visualization device 213 may ceasedisplay of the virtual model after the registration process. As such, insome examples, visualization device 213 may display the virtual guide ata different time (e.g., not contemporaneously) with the display of thevirtual model. As also discussed above, in some examples, visualizationdevice 213 may maintain the display of the virtual model after theregistration process and/or re-display the virtual model at a later timeduring the surgical procedure. As such, in some examples, visualizationdevice 213 may display the virtual guide at the same time (e.g.,contemporaneously) with the display of the virtual model.

The closed-loop tool control techniques described elsewhere in thisdisclosure may also be used in the context of total ankle arthroplasty.For instance, in one example, the closed-loop control may be applied tothe saw used in making the distal tibial cut, proximal calcaneus cut,the medial cut and the lateral cut, during the total ankle arthroplasty.In another example, the depth control techniques described elsewhere inthis disclosure may be applied in the context of a total anklearthroplasty to help ensure that the surgeon does not drill too deeply.For instance, as discussed above, the depth control techniques describedelsewhere in this disclosure may be applied to assist the surgeon whenperforming the tibial resection.

The surgical item tracking techniques described elsewhere in thisdisclosure with respect to shoulder joint repair surgery may also applyto total ankle arthroplasty to help healthcare professionals selectsurgical items and track use of the surgical items. Additionally, theworkflow management process described elsewhere in this disclosure maybe adapted for use in total ankle arthroplasty. For example, an XRvisualization device (e.g., XR visualization device 11702) may output anXR visualization (e.g., an MR visualization or VR visualization) thatincludes a set of virtual checklist items that correspond to items in achecklist of steps of a total ankle arthroplasty. For instance, insteadof virtual checklist items corresponding to the steps of FIG. 19, the XRvisualization may include virtual checklist items corresponding to stepsof FIG. 150 or any other steps of an ankle arthroplasty.

The collaboration and education techniques described elsewhere in thisdisclosure may also be used in the context of ankle surgery, such astotal ankle arthroplasty. For instance, in one example, a surgeon mayuse MR and/or VR to consult with a remote surgeon during a total anklearthroplasty or other type of ankle surgery. In another example,visualization device 213 may present an MR visualization to a surgeonthat includes a secondary view window that contains another person'sview of the patient's ankle, e.g., including both actual anatomicalobjects, i.e., the real patient anatomy, captured by a camera of theother person's visualization device and virtual objects generated as anMR visualization by the other person's visualization device. The rangeof motion tracking techniques described elsewhere in this disclosure mayalso be applied in the context of a patient's ankle instead of thepatient's shoulder.

In some examples, any of the techniques, devices, or methods describedherein may be used for medical and surgical educational purposes. Forexample, visualization devices that present mixed reality objects orinformation may be used to educate students about an orthopedic surgicalprocedure or one or more steps or stages of that procedure. The studentsbeing educated about the orthopedic surgical procedure may comprisephysicians being trained on the procedure or medical students beingtrained generally or specifically on the procedure. Alternatively, insome cases, a student being educated about the orthopedic medicalprocedure may be a patient on which the procedure will be performed, ora caretaker, guardian or family member of a patient. In this case, theteacher may comprise the surgeon or another medical professional thattrains the patient student. Also, medical technicians, nurses, physicianassistants, medical researchers, or any other person may be a studentaccording to the techniques and methods described herein. In someexamples, one or more teachers may provide illustrative instruction (andpossibly demonstrations) to one or more students though the use of mixedreality. Moreover, in still other examples, the student or students mayuse visualization devices to perform practice techniques or trainingtechniques of the orthopedic medical procedure with the aid of mixedreality. In still other cases, students and teachers may comprise apanel of experts working in a collaborative environment, in which case,student and teacher roles may change during a teaching session. Ingeneral, anyone being taught with the aid of mixed reality (or virtualreality) may be a student according to this disclosure, and similarly,anyone that teaches with the aid of mixed reality (or virtual reality)may be a teacher according to this disclosure.

FIG. 127 is a conceptual block diagram of an educational system 12701comprising an MR teacher device 12702 and an MR student device 12704. MRteacher device 12702 and MR student device 12704 may each comprise avisualization device 213, which is described in detail throughout thisdisclosure. MR teacher device 12702 may present the teacher with MReducational content for orthopedic surgery education 12706. Similarly,MR student device 12704 may present the student with similar MReducational content for orthopedic surgery education 12706. The MReducational content 12706, along with the tutelage of the teacherwearing MR teacher device 12702, may help to educate the student wearingMR student device 12704. In most cases, MR teacher device and MR studentdevice operate in the same physical location, although the techniques ofthis discloser are not limited in this respect. For example, it is alsopossible for MR teacher device 12702 and MR student device 12704 tooperate at remote physical locations relative to one another, in whichcase users may share MR educational content 12706 while viewingdifferent real-world backgrounds.

MR educational content 12706 may comprise one or more virtual elementsincluding a 3D virtual representation of one or more anatomical featuresassociated with the orthopedic surgical procedure. For example, the 3Dvirtual representation of the one or more anatomical features withineducational content 12706 may comprises a 3D virtual model of a humanshoulder, such as shown for example as 3D virtual models 1008, 1010 inFIG. 10. In some examples, the virtual representation of the one or moreanatomical features within educational content 12706 may comprise a 3Dvirtual illustration of a humeral head, a virtual illustration of ascapula, a 3D virtual illustration of a humeral bone or a 3D virtualillustration of a glenoid. In still other examples, the virtualrepresentation of the one or more anatomical features within educationalcontent 12706 may comprise a 3D virtual illustration of an ankle, avirtual illustration of a talus, or a 3D virtual illustration tibia or atibia head. Many educational details below are described in the contextof 3D virtual representations of shoulder anatomy, but the techniquesare also very useful for other anatomy, especially complex anatomicalelements, such as ankle anatomy.

Moreover, in addition to the 3D virtual representation of the one ormore anatomical features, educational content 12706 may further compriseadditional virtual elements demonstrate at least one aspect of theorthopedic surgical procedure. These additional elements, for examplemay comprise virtual pre-operative plan elements relative to the 3Dvirtual representation, one or more virtual surgical guidance featuresrelative to the 3D virtual representation, or one or more surgicalresults virtually illustrated on the 3D virtual representation so as todemonstrate at least one aspect of the orthopedic surgical procedure.

The users of educational system 12701 (e.g., a teacher and a studentwearing MR student device 12704 and MR teacher device 12702) may viewand manipulate virtual 3D elements through virtual controls, such as viagestures, gaze-based controls, voice inputs, or any control techniqueuseful in mixed reality or virtual reality. For example, the student orteacher may control virtual motion of a 3D virtual shoulder model andenable or disable virtual elements to demonstrate one or more aspect(pre-operative, inter-operative, and/or post-operative) of theorthopedic surgical procedure. Also, manual keypad input, touch screenentry, pointer controls, combinations of any virtual control mechanisms,or other types of controls may be used by teacher to manipulate virtual3D elements within educational content 12706.

In some cases, MR student device 12704 and/or MR teacher device 12702may include a haptic device that provides touch-based information to auser to help the users learn physical properties about virtual elementsand to manipulate virtual elements. The haptic device, for example, maycomprise one or more haptic gloves, one or more haptic wrist bands, ahaptic pen-type device, or another haptic device. The haptic device mayoperate with a visualization device to provide haptic feedback to a userand the haptic feedback may be associated with one or more victualelements presented to the user by the visualization device. Such hapticfeedback may be especially useful for surgical simulations performed byMR student device 12704. In this case, MR student device 12704 maycomprise a visualization device that presents one or more virtualelements and a haptic device that provide haptic feedback. In somecases, the haptic feedback may be synchronized or coordinated withmanipulations performed by the user on the one or more virtual elementspresented by the visualization device.

Using an MR device such as MR student device 12704 or MR teacher device12702, teachers or students may rotate, re-size, reposition, orotherwise move virtual 3D elements in space for educational reasons. MRstudent device 12704 or MR teacher device 12702 may enable or disableviewing of different segments of a virtual shoulder model, enable ordisable virtual elements showing virtual implants on a 3D virtual model,virtual surgical planning on the 3D virtual model, virtual surgicalguidance on the 3D virtual model, and/or virtual post-operative resultson the 3D model.

Moreover, teachers or students may show anatomical movement of boneswithin a shoulder socket or within a human ankle. The example teachermodel 1008 shown in FIG. 11 for example shows a 3D model of a shoulder,along with virtual elements showing a virtual 3D representation ofshoulder implant components 1010 and virtual elements showing a likelyimpingement point 1106 that may be caused by shoulder implantationcomponents 1010. Using an MR device such as MR student device 12704 orMR teacher device 12702, teachers or students may be able to rotate thevirtual humeral bone relative to the glenoid to show shoulder motion. Inaddition, teachers or students may also enable or disable viewing ofdifferent illustrated elements, which may be segmented. Portions ofhumeral bone may be segmented and selectively enabled or disabled, e.g.,to show a humeral cutting plane and a location for one or more shoulderimplant components on the humeral bone. Also, portions of a scapula orglenoid may be segmented and selectively enabled or disabled, e.g., toshow a location for one or more shoulder implant components on thescapula or glenoid.

The 3D virtual representations of one or more anatomical features may bebased on actual patient images or may be based on images of one or morepatients. The 3D virtual representations may be segmented into differentsub-components, which may be enabled or disabled by users. In somecases, the 3D virtual representations may be computer-generated. In somecases, the 3D virtual representations may be selected from a catalog of3D virtual representations (e.g., stored in memory of MR student device12704 or MR teacher device 12702 or stored remotely). The different 3Dvirtual representations in the catalog may demonstrate a wide variety ofdifferent shoulder conditions that may require orthopedic surgicalrepair. A teacher using MR teacher device 12702, for example, may selectone or more 3D virtual representations from the catalog of 3D images inorder to make educational demonstrations to a student wearing MR studentdevice 12704.

The 3D virtual representations may be selected for specific lesions bythe teacher to the student. MR teacher device 12702, for example maypresent a 3D shoulder model within MR educational content 12706 toillustrate a shoulder with a particular type of classification (e.g. atype of Walch classification), which may call for a particular type ofsurgical procedure or selection of particular implant components withparticular sizes, angles, and implant positions. For other lessons, MRteacher device 12702 may present different shoulder models havingdifferent classifications, thereby calling for a different type ofsurgical procedure or different implant components, sizes, angles and/orimplant positions.

In addition to 3D virtual representation of one or more anatomicalelements, such as a virtual of one or more anatomical elements, MReducational content 12706 may comprise any of a wide variety of MRcontent described herein, such as MR surgical guidance information, MRregistrational content, MR-based axes, planes or markers, MR-based jigsor guides, MR-based bone models or soft tissue models, MR-based guidanceon surgical tools, virtual implants, MR-based workflow checklists, rangeof motion information, pre-operative animations, or other MR-basedinformation. The particular MR content used in any given educationalsetting, however, may depend on the student that is being educated. Forexample, educational system 12701 may be used to educate or train aphysician, a medical student, a technician, a nurse, a physicianassistant, or any other person that may be involved in an orthopedicmedical procedure. Alternatively, educational system 12701 may be usedto educate a patient (or caretaker, guardian, patient family member,and/or patient friends) about a procedure to be performed. In stillother cases, educational system 12701 may be used to educate researchersor any other person that may have interests or reasons to learn one ormore details about an orthopedic surgical procedure. The MR educationalcontent for orthopedic surgery education 12706 may be selected ordefined for different educational settings.

MR educational content 12706 may comprise one or more virtual elementsthat include a 3D virtual representation of one or more anatomicalfeatures associated with the orthopedic surgical procedure. The 3Dvirtual representation of one or more anatomical features, for example,may comprise a 3D virtual model of a human shoulder, or possibly asegment of a human shoulder, such as a 3D virtual illustration of ahumeral head or a 3D virtual illustration of a glenoid. The virtualelements may be controllable by MR teacher device 12702 and/or MRstudent device 12704. MR teacher device 12702 may typically control thevirtual elements, but in some cases virtual control may be given to astudent wearing one of MR student devices 12704. Control of the virtualelements may be performed by the user of MR teacher device 12702 (or MRstudent devices 12704) with gestures, gazes, voice inputs, combinationsor gestures gazes or voice inputs, or other techniques used for MR or VRcontrol and manipulations of virtual content.

In some examples, MR educational content 12706 may comprise surgicalguidance information and this surgical guidance information may allowthe user of MR teacher device 12702 to train the user of MR studentdevice 12704 on the surgical procedure. In such cases, MR educationalcontent 12706 may comprise one or more virtual elements that include a3D virtual representation of one or more anatomical features, as well asvirtual guidance elements or information to guide a user on surgicalsteps shown or illustrated relative to the 3D virtual representation ofone or more anatomical features. In other examples, the MR educationalcontent 12706 may comprise one or more virtual elements positionedrelative to a physical (e.g., synthetic) anatomical model or anatomy ofa cadaver. Example physical anatomical models are commercially availablefrom Sawbones USA, Vashon Island, Wash., USA. Cadaver anatomy mayinclude an entire cadaver or cadaveric specimens.

As one example, MR educational content 12706 may comprise a virtualreaming axis positioned relative to a virtual 3D representation of aglenoid or relative to a physical model or a glenoid bone of a cadaver.In some cases, a virtual reaming contour may also be included in MReducational content 12706. As another example, MR educational content12706 may comprise a virtual cutting plane, such as a virtual humeralcutting plane shown relative to a 3D virtual representation of a humeralhead or a virtual cutting plane relative to a physical model of ahumeral bone or the humeral bone of a cadaver. In other examples, MReducational content 12706 may comprise a virtual jig or guide and mayillustrate placement of the virtual jig or guide relative to a 3Dvirtual representation of one or more anatomical features (such as avirtual glenoid) or positioned relative to a physical model or anatomyof a cadaver. In other examples, MR educational content 12706 mayillustrate a virtual drilling point or virtual drilling axis forinsertion of a guide post into a 3D virtual representation of a glenoid(or other virtual anatomical model) or for insertion of a guide postinto a physical anatomical model or the cadaver. In such examples, MReducational content 12706 may also illustrate a virtual axis relative tothe virtual jig or guide. In still other examples, MR educationalcontent 12706 may comprise virtual markers relative to a 3D virtualrepresentation of one or more anatomical features (e.g., a virtualmodel) or relative to a physical model or cadaver anatomy. The virtualmarkers may specify locations, points or axes for drilling, reaming,grinding, preparation for an implant, attachment of an implant, oranything that might be shown for intra-operative surgical guidance withrespect to anatomy of a patient. Indeed, any of the surgical guidancefeatures described elsewhere in this disclosure could also be presentedvirtually by MR teacher device 12702 and MR student device 12704relative to a virtual anatomical model (or a physical model or cadaver)to facilitate training on the surgical procedure.

The physical model may comprise a physical bone model, such as aphysical bone model commercially available from Sawbones USA or anothersource. With the aid of mixed reality, students may practice reaming orother surgical steps on the physical bone model. In other examples,virtual tissue models can be presented to students, allowing students toperform simulated surgical cuts or other manipulations on the virtualtissue models via MR student device 12704. Gestures by a user's figuresmay simulate cutting on virtual tissue models via MR student device12704 such that students can cut tissue, expose anatomical features(such as the glenoid) and place virtual implants or other componentsrelative to the virtual tissue model. Using gestures, such asfinger-based cutting gestures, students may manipulate virtual tissuemodels to expose layers of skin, fat, muscle, bone, or other anatomy.The virtual tissue model may be segmented, which may allow for differentlayers (skin, fat, muscle and bone) of the virtual tissue model to beshown, exposed and manipulated by students wearing MR student device12704 or by a teacher wearing MR teacher device 12702. MR student device12704 may perform finger tracking and hand tracking to illustratevirtual cuts or other manipulations on a virtual tissue model. Physicalmodels or virtual tissue models of human shoulders may be used forshoulder surgery education, and similarly, physical models or virtualtissue models of human ankles may be used for ankle surgery education.Other types of physical models or virtual tissue models (fingers,elbows, knees, back, neck, etc.) may also be used to promote orthopedicsurgical education with respect to other parts of the body.

When virtual elements are presented relative to physical models,cadavers, or virtual tissue models, it may be desirable to ensure thatthe virtual elements match the elements of the physical models,cadavers, or virtual models. Thus, it may be desirable to know theactual physical dimensions of the physical models, cadavers, or virtualtissue models, which may be obtained by scans or imaging. Accordingly,it may be desirable to have virtual elements that are generated ordefined based on scans or images of the physical models. This can ensurethat the virtual elements can be matched and properly registered to thephysical models, cadavers, or virtual models. Also, in a teachingenvironment, it may be useful to have many identical physical modelsthat have corresponding virtual elements that match the physical model.In this case, different students may perform actual manipulations ondifferent identical physical models, with the aid of virtual elementsand virtual guidance that is precisely defined based on the anatomy ofthe identical physical models.

In some examples, a system comprises a plurality of physical models ofan anatomical element, wherein the plurality of physical models aresubstantially identical, and a computer-readable medium comprisinginstructions that upon execution by a visualization device, cause thevisualization device to present virtual elements associated with ananatomical orthopedic surgical procedure relative to one of the physicalmodels, wherein the virtual elements include a virtual representation ofthe anatomical element that is substantially identical in size and shapeto the physical models.

In some examples, a system may include multiple physical models that arephysically used by different students or teachers with the aid of mixedreality. For example, a system may comprise a plurality of physicalmodels of an anatomical element wherein the plurality of physical modelsare substantially identical. The system may include a firstvisualization device configured to display a first mixed realitypresentation to a first user, wherein the first mixed realitypresentation includes one or more first virtual elements that arecontrollable by the first user while the first user is wearing the firstdevice and wherein the one or more first virtual elements comprise afirst 3D virtual representation of one or more anatomical featuresassociated with the orthopedic surgical procedure and wherein the first3D virtual representation is positioned relative to a first physicalmodel of the plurality of physical models. In addition, the system mayinclude a second visualization device configured to display a secondmixed reality presentation to a second user wherein the second mixedreality presentation includes one or more second virtual elements thatare controllable by the second user while the second user is wearing thesecond device and wherein the one or more second virtual elementscomprise a second 3D virtual representation of one or more anatomicalfeatures associated with the orthopedic surgical procedure and whereinthe second 3D virtual representation is positioned relative to a secondphysical model of the plurality of physical models. In this example, thefirst and second visualization devices may be used by students orteachers, and each user (the students and/or teachers) may have theirown physical model and their own mixed reality presentation shown withrespect to their own physical model. In some examples, physical modelsmay be created based on images or segmentations of actual patient orcadaver anatomical elements. In this case, the physical models may be 3Dprinted to generate a plurality of identical 3D models, and virtualelements can be defined based on the same images or segmentations of theactual patient or cadaver anatomical elements. Students and teachers canperform trial surgical steps on their own physical model (that is a 3Drepresentation of actual patient or cadaver anatomy) and mixed-realityvirtual elements or virtual models can be presented relative to thedifferent physical models by visualization devices worn by the studentsand teachers.

In these and other examples, MR educational content 12706 may be usefulfor training a physician on how to perform steps of an orthopedic jointrepair procedure, such as any of a variety shoulder surgery procedures(e.g., such as an anatomical or reverse shoulder arthroplasty procedure)or any of a variety of ankle surgery procedures (e.g., such as an anklearthroplasty procedures). Accordingly, in these examples, the studentwearing MR student device 12704 would typically be a surgeon, althoughother students could also benefit from the MR-based education.

In other examples, MR educational content 12706 may comprise virtualcontent illustrative of a registration process for registering a virtualmodel to an actual anatomical feature of a patient, a physical bone orsoft tissue model, or a cadaver feature. For example, MR teacher device12702 and MR student device 12704 may be configured to perform aregistration process for registering a virtual model to an actualanatomical feature, e.g., registering a virtual model of a glenoid to anactual glenoid or a physical model, such as from a cadaver or asynthetic bone model, respectively. In this case, a teacher wearing MRteacher device 12702 may comprise a medical device technician. Toregister a virtual model on an actual glenoid mode or a physical model,the teacher may instruct a student (e.g., a surgeon) wearing MR studentdevice 12704 on the registration steps of “SET,” “MATCH,” and “RESET,”described in greater detail elsewhere in this disclosure for registeringa virtual model to an actual anatomical feature of a patient.

In still other examples, MR educational content 12706 may comprisevirtual trialing information, e.g., to compare a virtual model of reamedbone to a virtual model of an implant to see whether the virtual modelof the reamed bone has been shaped properly to receive an implant. Inother words, MR educational content 12706 may include virtual elementsin the form of a virtual model of implant components that can bepositioned relative to a 3D virtual model (e.g., a virtual glenoid) orrelative to a physical model of the glenoid or a cadaver to see whetherthe virtual implant fits on the virtual reamed glenoid bone, physicalmodel, or cadaver bone. In this case, a teacher wearing MR teacherdevice 12702 may instruct a student wearing MR student device 12704 onthe reaming process. In some cases, the student can view a virtualimplant relative to a virtual model of reamed bone. In other cases, thestudent may actually perform the reaming process on a cadaver glenoidbone or a model glenoid bone, e.g., using virtual intra-operativeguidance provided via MR student device 12704, and then MR studentdevice 12704 and MR teacher device 12702 can present a virtual model ofthe implant. The student or teacher can then manipulate the position ofthe virtual model of the implant relative to the reamed bone or reamedmodel to see whether the implant fits properly. This process can provideuseful training to the student on the reaming process of a shouldersurgery, as one example of an orthopedic surgical procedure task, and inthis example, student may comprise a medical student, a surgeon, or anyperson being trained on the surgical procedure. Similar training mayalso be used for ankle procedures or other relatively complex orthopedicprocedures, allowing students to practice an MR-guided procedure orprocedure step on the talus and/or tibia of a cadaver.

In other examples, MR educational content 12706 may comprise informationto aid in a registration process of a depth aid element, such asdescribed in greater detail elsewhere in this disclosure. For example,as described in detail elsewhere in this disclosure, a depth aid elementmay be used to aid in depth calculations of a tooling bit, such as areaming element, a drilling element or other another tooling bit. Theregistration process shown and described, for example, with reference toFIGS. 80-82 may be implemented by MR student device 12704 and MR teacherdevice 12702 in order to train a student (e.g., a surgeon) on how toperform the registration process. In some cases, the teacher wearing theMR teacher device 12702 may comprise a medical device technician thatmay train surgeons on how to perform the registration process on thedepth aid element.

In other examples, MR educational content 12706 may comprise virtualinformation on surgical tool or implant component selection or virtualaids, such as virtual elements that identify tools or implant componentsfor use in an orthopedic surgical procedure. Indeed, this disclosuredescribes many examples of virtual elements that can be presented by avisualization device in order to aid nurses or other surgicalparticipants with tool selection. Such techniques and tools may also beuseful in educational system 12701. In such cases, MR student device12704 may be worn by a nurse or medical assistant, and MR teacher device12702 may be worn by a person training the nurse on the surgicalprocedure and tool selection. MR student device 12704 and MR teacherdevice 12702 may present MR visualizations for nurse training that aresimilar to those described elsewhere in this disclosure for use by thenurse in the actual surgical procedure. Educational training on thesurgical procedure, including tool selection training for the nurse, mayhelp to improve the surgical procedure.

In some examples, MR educational content 12706 may relate to planning ofthe surgical procedure and may include implant components, size,positions, angles, reaming axis, reaming contour, cutting plane, orother features so that a student can visualize reaming of the glenoidand placement of a particular implant with cutting of the humeral boneand placement of a humeral implant, with selected sizes, positions,angles, (e.g., for different procedures such as anatomical or reverseshoulder arthroplasty). The student wearing MR student device 12704 maypractice planning a surgical procedure for a particular virtual shouldermodel having a particular type of problem, and the teacher wearing MRteacher device 12702 may train the student on multiple differentshoulder models representing different shoulder problems, differentshoulder classifications, and/or different types of surgery to beplanned.

In other examples, MR educational content 12706 may comprise virtualtraining information or virtual visual aids on the use of automatedtools that includes closed loop control. For example, automated toolshave been described herein for use in an orthopedic surgical procedurewhereby the tools can be automatically enabled or disabled based on theuse of the tool. As one example, closed loop-controlled tools have beendescribed that may automatically disable once the tool has performed itsfunction (such as a reamer that is disabled once a desired reaming depthis achieved). Educational training on the usage of such tools can behelpful to a surgeon. Accordingly, in some examples, MR student device12704 and MR teacher device 12702 may present virtual elements thatillustrate or demonstrate tool usage, as well as tool disabling once thetool has performed its function.

In still other examples, MR educational content 12706 may comprise MRsurgical workflow guidance information, such as a step-by-step workflowor a checklist that is presented by MR student device 12704 and MRteacher device 12702. A student wearing MR student device 12704 maywatch a virtual workflow that is presented as a surgeon wearing MRteacher device 12702 performs the procedure. For instance, MR studentdevice 12702 may generate an MR visualization that contains virtualchecklist items. In some cases, the virtual workflow presented to thestudent wearing MR student device 12704 may be different (possibly moreconspicuous) than that presented to the surgeon wearing MR teacherdevice 12702. This can allow the student to be well informed andeducated about the surgical procedure, as the procedure occurs. In thisexample, the student may be a medical student or possibly a surgeon(i.e., a student surgeon) that is being trained by another surgent(i.e., a teacher surgeon).

In other examples, MR educational content 12706 may comprise virtualinformation about range of motion. In this case, a physician or surgeonwearing MR teacher device 12702 may educate a patient wearing MR studentdevice 12704 by showing virtual demonstrations of range of motioninformation associated with an orthopedic surgical procedure, such as ashoulder surgery. Also, MR educational content 12706 comprising range ofmotion information may also be useful for training surgeons, medicalstudents, or others on the range of motion affects associated with asurgical step. For example, MR educational content 12706 may compriserange of motion information indicative or range of motion (or loss ofrange of motion) associated with the implantation of a particularimplant. If, for example, an incorrectly sized or positioned implant isimplanted in a patient, this may have negative consequences on range ofmotion. MR teacher device 12702 and MR student device 12704 may presentrange of motion visualizations showing the likely range of motionconsequences associated with a particular implant. In this case, forexample, MR educational content 12706 may comprise range of motioninformation or range of motion demonstrations that virtually illustratepossible impingement points over a desired range of motion, where theimpingements may be caused by an incorrectly sized or positionedimplant. In some cases, MR educational content 12706 may include virtualmarkers of likely impingement points due to placement of an implant in apatient.

In yet other examples, MR educational content 12706 may comprise virtualpre-operative animations, e.g., which may show desirable results orproblems associated with any given implantation or implantationprocedure. For example, MR teacher device 12702 and MR student device12704 may present virtual animations showing possible impingementsassociated with the implantation of an incorrectly sized implant.Similarly, MR teacher device 12702 and MR student device 12704 maypresent virtual animations showing a desirable outcome associated withthe implantation of a correctly sized implant. In some cases, MR teacherdevice 12702 and MR student device 12704 may present instant virtualfeedback to the teacher and the student, which demonstrates orillustrates ramifications (e.g., desirable outcomes, or undesirableimpingements or loss of range of motion) associated with a particularimplant or a particular implantation procedure.

Educational system 12701 of FIG. 127 is one example of a system that candemonstrate at least one aspect of an orthopedic surgical procedure. Afirst device (e.g., MR teacher device 12702) can be configured todisplay a presentation to a first user (i.e., a teacher), wherein thepresentation includes one or more virtual elements that are controllableby the first user while the first user is wearing the first device,wherein the one or more virtual elements comprise a 3D virtualrepresentation of one or more anatomical features associated with theorthopedic surgical procedure. A second device (e.g., MR student device12704) may also be configured to display the presentation to a seconduser, wherein the one or more virtual elements further include one ormore virtual pre-operative plan elements relative to the 3D virtualrepresentation, one or more virtual surgical guidance features relativeto the 3D virtual representation, one or more surgical results virtuallyillustrated on the 3D virtual representation so as to demonstrate atleast one aspect of the orthopedic surgical procedure. Demonstrating atleast one aspect of the orthopedic surgical procedure, for example, maycomprise presenting virtual pre-operative plan elements relative to the3D virtual representation, presenting one or more virtual surgicalguidance features relative to the 3D virtual representation, and/orpresenting one or more surgical results virtually illustrated on the 3Dvirtual representation so as to demonstrate at least one aspect of theorthopedic surgical procedure.

The one or more virtual elements that are controllable by the first userwhile the first user is wearing the first device and viewable by thesecond user while the second user is wearing the second device mayfacilitate education on an orthopedic surgical procedure. MR educationalcontent 12706 may comprise the one or more virtual elements that arecontrollable by the first user while the first user is wearing the firstdevice (e.g., MR teacher device 12702) and viewable by the second userwhile the second user is wearing the second device (e.g., MR studentdevice 12704). The one or more virtual elements may include a 3D virtualrepresentation of one or more anatomical features associated with theorthopedic surgical procedure. In some examples, the one or more virtualelements may illustrate a virtual cutting axis relative to the 3Dvirtual representation. In some examples, the one or more virtualelements may illustrate a virtual reaming axis relative to the 3Dvirtual representation. In some examples, the one or more virtualelements may illustrate a virtual drilling axis relative to the 3Dvirtual representation. In some examples, the one or more virtualelements may illustrate placement of a virtual jig or guide relative to3D virtual representation. In some examples, the one or more virtualelements may illustrate a virtual axis relative to a virtual jig orguide. In some examples, the one or more virtual elements may comprisesurgical guidance information presented relative to the 3D virtualrepresentation. In some examples, the one or more virtual elements mayillustrate a registration process for registering the 3D virtualrepresentation to a physical model or a corresponding feature of acadaver. In some examples, the one or more virtual elements may includetrialing information associated with a prepared implantation locationfor implant component. In some examples, the one or more virtualelements may illustrate a registration process for registering a depthaid element. In some examples, the one or more virtual elements maycomprise virtual training information or virtual visual aids on the useof automated tools that include closed loop control. In some examples,the one or more virtual elements may comprise virtual information aboutrange of motion. In some examples, the one or more virtual elements maycomprise a virtual pre-operative animation. In some examples, the one ormore virtual elements may illustrate one or more virtual implantcomponents relative to the 3D virtual representation.

FIG. 128 is a conceptual block diagram of an educational system 12801comprising an MR teacher device 12802 and a plurality of MR studentdevices 12804A and 12804B through 12804N (collectively student devices12804). MR teacher device 12802 and MR student devices 12804 may eachcomprise a visualization device 213, which is described in detailthroughout this disclosure. MR teacher device 12802 may present theteacher with MR educational content for orthopedic surgery education12806. Similarly, MR student devices 12804 may present the student withsimilar MR educational content for orthopedic surgery education 12806.The MR educational content 12806, along with the tutelage of the teacherwearing MR teacher device 12802, may help to educate the student wearingMR student device 12804.

MR educational content 12806 may comprise content that is similar oridentical to MR educational content 12706 shown and described in FIG.127. Moreover, MR educational content 12806 may comprise any of the MRcontent disclosed elsewhere in this disclosure, which may also be usedfor surgical guidance, surgical planning, post-operative analysis orother reasons. Indeed, MR educational content 12806 may comprise any ofa wide variety of MR content described herein. In some cases, MR studentdevices 12804 and/or MR teacher device 12802 may comprise both avisualization device that presents virtual elements to a user and ahaptic device that provides touch-based information to a user.

As with other examples, the actual MR content used in any giveneducational setting, however, may depend on the student that is beingeducated. For example, educational system 12801 may be used to educateor train physicians, medical students, technicians, nurses, physicianassistants, or any other persons (such as patients, caregivers,guardians, family or others) that may be involved in an orthopedicmedical procedure. Alternatively, educational system 12801 may be usedto educate a patients, family and friends about a procedure to beperformed on a patient. In still other cases, educational system 12801may be used to educate researchers or any other person that may haveinterests or reasons to learn one or more details about an orthopedicsurgical procedure. According to FIG. 128, multiple students associatedwith MR student devices 12804 may benefit from the teaching of a teacherusing MR teacher device 12802.

Educational system 12801 of FIG. 128 is another example of a system thatcan demonstrate at least one aspect of an orthopedic surgical procedure.A first device (e.g., MR teacher device 12802) can be configured todisplay a presentation to a first user (i.e., a teacher), wherein thepresentation includes one or more virtual elements that are controllableby the first user while the first user is wearing the first device andwherein the one or more virtual elements comprise a 3D virtualrepresentation of one or more anatomical features associated with theorthopedic surgical procedure. A second device (e.g., one of MR studentdevices 12804) may also be configured to display the presentation to asecond user. The one or more virtual elements may further include one ormore virtual pre-operative plan elements relative to the 3D virtualrepresentation, one or more virtual surgical guidance features relativeto the 3D virtual representation, or one or more surgical resultsvirtually illustrated on the 3D virtual representation so as todemonstrate at least one aspect of the orthopedic surgical procedure.

For shoulder surgery educational examples, the virtual representation ofthe one or more anatomical features that forms part of MR educationalcontent 12806 may comprise a 3D virtual model of a human shoulder. Asanother example, the virtual representation of the one or moreanatomical features that forms part of MR educational content 12806 maycomprise a 3D virtual illustration of a humeral head or a 3D virtualillustration of a glenoid. However, for other types of orthopedicsurgery educational examples, different types of virtual 3D models ofanatomical features may be presented in MR educational content 12806.For example, ankle surgery education may involve the presentation of a3D virtual illustration of a human ankle or 3D virtual illustrations ofthe talus and/or the tibia.

One or more virtual elements that are controllable by the first userwhile the first user wearing the first device and viewable by the seconduser while the second user is wearing the second device may facilitateeducation on an orthopedic surgical procedure. MR educational content12806 may comprise the one or more virtual elements that arecontrollable by the first user while the first user is wearing the firstdevice (e.g., MR teacher device 12802) and viewable by the second userwhile the second user is wearing the second device (e.g., one of MRstudent devices 12804). The one or more virtual elements may include anyof those described above (or combinations of those described above) withregard to MR educational content 12706 of FIG. 127.

FIG. 129 is a conceptual block diagram of an educational system 12901.Educational system 12901 comprises an MR/VR student device 12904, whichmay each comprise a mixed reality device, such as visualization device213 described in detail throughout this disclosure, or a virtual realitydevice that presents only virtual information and no real-world views.MR/VR student device 12904 may present the student with MR/VR (mixedreality or virtual reality) educational content for orthopedic surgeryeducation 12906. In this example, however, the MR/VR educational content12906 includes an avatar teacher 12902.

In the example of FIG. 129, MR/VR educational content may comprisepre-recorded content presented by a pre-recorded avatar teacher 12902.The user of MR student device 12904 may select content from a menu orlist and may view the content via MR student device 12904. In somecases, MR student device 12904 may include a visualization device thatpresents virtual elements to a user and a haptic device that providestouch-based information to the user.

Similar to MR educational content 12706 of FIG. 127 and MR educationalcontent 12806 of FIG. 128, MR/VR educational content 12906 shown in FIG.129 may comprise any of the virtual content disclosed elsewhere in thisdisclosure. For example, MR/VR educational content 12906 may compriseany of a wide variety of content described herein, such as one or morevirtual elements including a 3D virtual representation of one or moreanatomical features associated with the orthopedic surgical procedure.In addition to the 3D virtual representation (e.g., a virtual model of ahuman shoulder, a virtual model of a glenoid or glenoid surface, or avirtual model of a humeral bone or humeral head), MR/VR educationalcontent 12906 may comprise additional virtual content. This additionalvirtual content may include any of the content virtual content orvirtual elements (or combinations of such virtual elements) describedabove with regard to MR educational content 12706 of FIG. 127.

As with other examples, the actual content used in any given educationalsetting, however, may depend on the student that is being educated. Forexample, educational system 12901 may be used to educate or trainphysicians, medical students, technicians, nurses, physician assistants,or any other persons (such as patients, caregivers, guardians, family orothers) that may be involved in an orthopedic medical procedure. In manycases, however, educational system 12901 may be useful to educate apatients, family and friends about a procedure to be performed on apatient. For example, MR/VR educational content 12906 may comprisevirtual information stored on a server that is accessible to userswearing MR/VR student device 12904. Pre-recorded orthopedic surgerycontent may be especially useful for educating non-expert users, such aspatients, family and friends of a patient.

FIG. 130 is a conceptual block diagram of an educational system 13001,which may be useful for a remote teaching environment. In the exampleshown in FIG. 130, the teacher may be located remotely relative to thestudents, and the teacher may wear VR teacher device 13002, which maycomprise a virtual reality device. Students may interact with theteacher via MR/VR student devices 13004A and 13004B through 13004N(collectively MR/VR student devices 13004).

MR/VR student devices 13004 may each comprise a mixed reality device,such as visualization device 213 described in detail throughout thisdisclosure, or a virtual reality device that presents only virtualinformation and no real-world views. MR/VR student devices 13004 maypresent each student with MR/VR (mixed reality or virtual reality)educational content for orthopedic surgery education 13006. Similar toMR educational content 12706 of FIG. 127 and MR educational content12806 of FIG. 128, and MR/VR educational content 12906 of FIG. 128,MR/VR educational content 13006 of FIG. 130 may comprise any of thevirtual content disclosed elsewhere in this disclosure. In some cases,MR/VR student devices 13004 and/or VR teacher device 13002 may compriseboth a visualization device that presents virtual elements to a user anda haptic device that provides touch-based information to that same user.

MR/VR educational content for orthopedic surgery education 13006 maycomprise a 3D virtual representation of one or more anatomical featuresassociated with the orthopedic surgical procedure. In addition to the 3Dvirtual representation (e.g., a virtual model of a human shoulder, avirtual model of a glenoid or glenoid surface, or a virtual model of ahumeral bone or humeral head), MR/VR educational content 13006 maycomprise additional virtual content. This additional virtual content mayinclude one or more additional virtual elements such as any of thosedescribed above (or combinations of those described above) with regardto MR educational content 12706 of FIG. 127.

As with other examples, the actual content used in any given educationalsetting, however, may depend on the student that is being educated. Forexample, educational system 13001 may be used to educate or trainphysicians, medical students, technicians, nurses, physician assistants,or any other persons (such as patients, caregivers, guardians, family orothers) that may be involved in an orthopedic medical procedure.Moreover, the use of virtual reality (e.g., VR teacher device 13002) canallow the teacher to located remotely relative to the students. In thisexample, the students may use either mixed reality or virtual reality(e.g., MR/VR student devices 13004) to interact with the teacher wearingVR teacher device 13002 and to view MR/VR educational content 13006,which may be selected, presented or otherwise assigned to the studentsby the teacher.

Educational system 13001 of FIG. 130 is another example of a system thatcan demonstrate at least one aspect of an orthopedic surgical procedure.A first device (e.g., VR teacher device 13002) can be configured todisplay a presentation to a first user (i.e., a teacher), wherein thepresentation includes one or more virtual elements that are controllableby the first user wearing the first device and wherein the one or morevirtual elements comprise a 3D virtual representation of one or moreanatomical features associated with the orthopedic surgical procedure. Asecond device (e.g., one of MR/VR student devices 13004) may also beconfigured to display the presentation to a second user. The one or morevirtual elements may further include one or more virtual pre-operativeplan elements shown relative to the 3D virtual representation, one ormore virtual surgical guidance features shown relative to the 3D virtualrepresentation, or one or more surgical results virtually illustrated onthe 3D virtual representation so as to demonstrate at least one aspectof the orthopedic surgical procedure.

The one or more virtual elements that are controllable by the first userwearing the first device and viewable by the second user wearing thesecond device may facilitate education on an orthopedic surgicalprocedure. For example, MR/VR educational content 13006 may comprise anyof a wide variety of content described herein and may include a 3Dvirtual representation of one or more anatomical features associatedwith the orthopedic surgical procedure. In addition to the 3D virtualrepresentation (e.g., a virtual model of a human shoulder, a virtualmodel of a glenoid or glenoid surface, or a virtual model of a humeralbone or humeral head), MR/VR educational content 13006 may compriseadditional virtual content. This additional virtual content may includeany of the virtual elements (or combinations) described with regard toMR educational content 12706 of FIG. 127.

FIG. 131 is another conceptual block diagram of an educational system,e.g., educational system 13101, which may be useful for a remoteteaching environment. In this example, a teacher may wear MR teacherdevice 13102 to present MR/VR educational content 13106 to variousstudents. A first student may wear MR student device 13104A, such asvisualization device 213 described in detail throughout this disclosure.The first student and the teacher may be physically located in the sameroom to facilitate a shared MR experience. A second student, however,may be located remotely relative to the teacher and may wear VR studentdevice 13104B to participate in the teaching exercise. VR student device13104B may present only virtual information and no real-world views. Insome cases, the teacher and the other students (e.g., the teacherwearing MR teacher device 13102 and the first student wearing MR studentdevice 13104A) may be presented as avatars in the VR presentation of VRstudent device 13104B, giving the perception that the students andteacher are all located in the same room even though the studentassociated with VR student device 13104B is located remotely. Similarly,the second student may also be presented as an avatar to the teacherwearing MR teacher device 13102 and the first student wearing MR studentdevice 13104A. FIG. 131 also illustrates MR/VR student device N 13104N,which generally means that any number of MR or VR participants may bepresent in the educational session. In some cases, MR devices may beused by participants that are located remotely relative to one another,in which case, users may share views of virtual elements without sharingthe same real-world views.

Like the examples shown in FIGS. 127, 128, 129 and 130, in the exampleof FIG. 131, MR/VR educational content 13106 may comprise any of thevirtual content disclosed elsewhere in this disclosure such as one ormore virtual elements including a 3D virtual representation of one ormore anatomical features associated with the orthopedic surgicalprocedure. In addition to the 3D virtual representation (e.g., a virtualmodel of a human shoulder, a virtual model of a glenoid or glenoidsurface, or a virtual model of a humeral bone or humeral head), MR/VReducational content 13006 may comprise additional virtual content. Thisadditional virtual content may include one or more virtual elementspresented relative to the 3D virtual representation and may include anyof those described above (or combinations of those described above) withregard to MR educational content 12706 of FIG. 127. Also, in some cases,student devices 13104 and/or MR teacher device 13102 may comprise both avisualization device that presents virtual elements to a user and ahaptic device that provides touch-based information to the user.

As with other examples, the actual content used in any given educationalsetting, however, may depend on the student that is being educated. Forexample, educational system 13101 may be used to educate or trainphysicians, medical students, technicians, nurses, physician assistants,or any other persons (such as patients, caregivers, guardians, family orothers) that may be involved in an orthopedic medical procedure.Moreover, the use of virtual reality (e.g., VR student device 13104B)can allow one or more of the students to be located remotely relative tothe teacher, who in this example, wears MR teacher device MR teacherdevice 13102. At the same time, however, the teacher may be able to viewand interact with any local students, such as a student wearing MRstudent device 13104B.

Educational system 13101 of FIG. 131 is another example of a system thatcan demonstrate at least one aspect of an orthopedic surgical procedure.A first device (e.g., MR teacher device 13102) can be configured todisplay a presentation to a first user (i.e., a teacher), wherein thepresentation includes one or more virtual elements that are controllableby the first user wearing the first device and wherein the one or morevirtual elements comprise a 3D virtual representation of one or moreanatomical features associated with the orthopedic surgical procedure. Asecond device (e.g., MR student device 13104A or VR student device13104B) may also be configured to display the presentation to a seconduser. The one or more virtual elements may further include one or morevirtual pre-operative plan elements shown relative to the 3D virtualrepresentation, one or more virtual surgical guidance features shownrelative to the 3D virtual representation, or one or more surgicalresults virtually illustrated on the 3D virtual representation so as todemonstrate at least one aspect of the orthopedic surgical procedure.

The one or more virtual elements that are controllable by the first userwearing the first device and viewable by the second user wearing thesecond device may facilitate education on an orthopedic surgicalprocedure. MR educational content 13106 may comprise the one or morevirtual elements that are controllable by the first user wearing thefirst device (e.g., MR teacher device 13102) and viewable by the seconduser wearing the second device (e.g., MR student device 13104A or MRstudent device 13104B).

FIGS. 132 and 133 are conceptual block diagrams of other educationalsystems 13201 and 13301 that use mixed reality and virtual reality fororthopedic surgical education. In the example of FIG. 132, multipleteachers wear MR/VR teacher devices 13202A and 13202B through 13202N(collectively MR/VR teacher devices 13202). Moreover, in the example ofFIG. 132, multiple students wear MR/VR student devices 13204A and 13204Bthrough 13204N (collectively MR/VR student devices 13204). In thisexample, any of the participants (teachers or students) may participatein the teaching session using mixed reality or virtual reality. That isto say, any of MR/VR teacher devices 13202 and MR/VR student devices13204 may comprise mixed reality devices such as visualization device213, or alternatively, MR/VR teacher devices 13202 and MR/VR studentdevices 13204 may comprise virtual reality devices the present onlyvirtual information, in which case real-world information presented tousers wearing mixed reality devices may be presented as virtualinformation to users wearing virtual reality devices. Moreover, MR userswearing mixed reality devices may be presented as avatars to VR userswearing virtual reality devices, and VR users wearing virtual realitydevices may be presented as avatars to MR users wearing mixed realitydevices.

The example of FIG. 133 is somewhat similar to that of FIG. 132. In theexample of FIG. 133, a first teacher wears MR teacher device 13302A anda second teacher wears VR teaching device 13302B. Moreover, in theexample of FIG. 133, a first student wears MR student device 13304A anda second student wears VR student device 13304B. Additional teachers orstudents may also be present using mixed reality or virtual reality, asgenerally illustrated by MR/VR teacher device N 13302N and MR/VR studentdevice N 13304N. In general, any MR device (e.g., MR teacher device13302A and MR student device 13304B) uses mixed reality whereas any VRdevice (e.g., VR teacher device 13302B or VR student device 13304 usesvirtual reality).

According to the example shown in FIG. 133, real-world information orobjects viewed by users wearing mixed reality devices (such as MRteacher device 13302A and MR student device 13304A) may be presented asvirtual information to users wearing virtual reality devices (such as VRteacher device 13302B and VR student device 13304B). For example, a userof MR teacher device 13302A may view a real-world object (such as acadaver) and that real-world object may be presented to VR devices (suchas VR teacher device 13302B and VR student device 13304B) as a virtualobject. Moreover, MR users wearing mixed reality devices may bepresented as avatars to VR users wearing virtual reality devices, and VRusers wearing virtual reality devices may be presented as avatars to MRusers wearing mixed reality devices.

Similar to the other examples shown in FIGS. 127, 128, 129, 130, and131, in the examples of FIGS. 132 and 133, MR/VR educational content13206 or 13306 may comprise any of the virtual content disclosedelsewhere in this disclosure. For example, MR/VR educational content13206 may comprise any of a wide variety of content described herein,such as one or more virtual elements including a 3D virtualrepresentation of one or more anatomical features associated with theorthopedic surgical procedure. In addition to the 3D virtualrepresentation (e.g., a virtual model of a human shoulder, a virtualmodel of a glenoid or glenoid surface, or a virtual model of a humeralbone or humeral head), MR/VR educational content 13206 may compriseadditional virtual content. This additional virtual content may includeany of the virtual elements described above (or combinations of thosedescribed above) with regard to MR educational content 12706 of FIG.127. Also, in some cases, student devices 13304, 13404 and/or teacherdevices 13302, 13402 may each comprise both a visualization device thatpresents virtual elements to a user and a haptic device that providestouch-based information to the user.

Again, the actual content used in any given educational setting,however, may depend on the student that is being educated. For example,educational system 13201 or 13301 may be used to educate or trainphysicians, medical students, technicians, nurses, physician assistants,or any other persons (such as patients, caregivers, guardians, family orothers) that may be involved in an orthopedic medical procedure.Moreover, the use of virtual reality (e.g., VR teacher device 13302B orVR student device 13304B) can allow one or more of the students andteachers to be located remotely relative to one another. At the sametime, any users of mixed reality (e.g., MR teacher device 13302A or MRstudent device 13304A) may be able to view and interact with other localparticipants using mixed reality.

Educational system 13201 of FIG. 132 is another example of a system thatcan demonstrate at least one aspect of an orthopedic surgical procedure.A first device (e.g., one of MR/VR teacher devices 13202) can beconfigured to display a presentation to a first user (i.e., a teacher),wherein the presentation includes one or more virtual elements that arecontrollable by the first user wearing the first device and wherein theone or more virtual elements comprise a 3D virtual representation of oneor more anatomical features associated with the orthopedic surgicalprocedure. A second device (e.g., one of MR/VR student devices 13204)may also be configured to display the presentation to a second user. Theone or more virtual elements may further include one or more virtualpre-operative plan elements shown relative to the 3D virtualrepresentation, one or more virtual surgical guidance features shownrelative to the 3D virtual representation, or one or more surgicalresults virtually illustrated on the 3D virtual representation so as todemonstrate at least one aspect of the orthopedic surgical procedure.

The one or more virtual elements that are controllable by the first userwearing the first device and viewable by the second user wearing thesecond device may facilitate education on an orthopedic surgicalprocedure. MR educational content 13206 may comprise the one or morevirtual elements that are controllable by the first user wearing thefirst device (e.g., one of MR/VR teacher devices 13202) and viewable bythe second user wearing the second device (e.g., one of MR/VR studentdevices 13204).

Educational system 13301 of FIG. 133 is another example of a system thatcan demonstrate at least one aspect of an orthopedic surgical procedure.A first device (e.g., MR teacher device 13302A or VR teacher device13302B) can be configured to display a presentation to a first user(i.e., a teacher), wherein the presentation includes one or more virtualelements that are controllable by the first user wearing the firstdevice and wherein the one or more virtual elements comprise a 3Dvirtual representation of one or more anatomical features associatedwith the orthopedic surgical procedure. A second device (e.g., MRstudent device 13304A or VR student device 13304B) may also beconfigured to display the presentation to a second user. The one or morevirtual elements may further include one or more virtual pre-operativeplan elements shown or illustrated relative to the 3D virtualrepresentation, one or more virtual surgical guidance features shown orillustrated relative to the 3D virtual representation, or one or moresurgical results virtually illustrated on the 3D virtual representationso as to demonstrate at least one aspect of the orthopedic surgicalprocedure.

The one or more virtual elements that are controllable by the first userwearing the first device and viewable by the second user wearing thesecond device may facilitate education on an orthopedic surgicalprocedure. MR educational content 13306 may comprise the one or morevirtual elements that are controllable by the first user wearing thefirst device (e.g., MR teacher device 13302A or VR teacher device13302B) and viewable by the second user wearing the second device (e.g.,MR student device 13304A or VR student device 13304). In variousexamples, the one or more virtual elements may include any of thevirtual elements any of those described above (or combinations of thosedescribed above) with regard to MR educational content 12706 of FIG.127.

FIG. 134 is another conceptual block diagram of an educational system,e.g., educational system 13401 that uses mixed reality and/or virtualreality for orthopedic surgical education. In the example of FIG. 134, ateacher wears MR/VR teacher device 13402 and multiple students wearMR/VR student devices 13404A and 13404B through 13404N (collectivelyMR/VR student devices 13404). In this example, any of the participants(teacher or students) may participate in the teaching session usingmixed reality or virtual reality. That is to say, any of MR/VR teacherdevice 13402 and MR/VR student devices 13404 may comprise mixed realitydevices such as visualization device 213, or alternatively, MR/VRteacher device 13402 and MR/VR student devices 13404 may comprisevirtual reality devices that present only virtual information, in whichcase real-world information presented to users wearing mixed realitydevices may be presented as virtual information to users wearing virtualreality devices and the virtual information may be based on real worldobjects viewed by users of MR devices. MR/VR educational content 13406may comprise any of the virtual content disclosed elsewhere in thisdisclosure, such as any of the educational content described withrespect to FIGS. 127-133 or elsewhere in this disclosure. In addition,with the example shown in FIG. 134, MR/VR educational content 13406 mayinclude multiple different virtual models to be used and manipulated bythe students and the teacher, which are shown in FIG. 13408 as teachermodel 13408 and student model 13405A and 13405B through 13405N(collectively referred to as student models 13405). The separate virtualmodels for students and teachers may comprise virtual content, such asstudent and teacher copies of virtual models (e.g., student and teachercopies of 3D virtual models of a human shoulder, student and teachercopies of 3D virtual models of a segment of a human shoulder, studentand teacher copies of a 3D virtual illustration of a humeral head, orstudent and teacher copies of a 3D virtual illustration of a glenoid).In addition, virtual student and teacher content may include additionalvirtual elements presented relative to such virtual models 13408 and13405.

Educational system 13401 may be useful in demonstrating at least oneaspect of an orthopedic surgical procedure. System 13401 may comprise afirst device (e.g., MR/VR teacher device 13402) configured to display apresentation to a first user wherein the presentation includes teachermodel 13408 in the form of a 3D virtual illustration of an anatomicalelement, wherein teacher model 13408 is controllable by the first user(i.e., the teacher) wearing the first device. System 13401 may furthercomprise a second device (e.g., one of MR/VR student devices 13404)configured to display the presentation to a second user (e.g., one ofthe students) wherein the presentation also one of student models 13405in the form of an additional 3D virtual illustrations of the anatomicalelement, wherein the student model (one of models 13405) is controllableby the second user (i.e., one of the students) wearing the first device.

In addition to the 3D virtual illustrations of an anatomical element,teacher model 13408 may include additional virtual elements (such as onemore of those described above with regard to FIGS. 127-133) todemonstrate at least one aspect of the orthopedic surgical procedure.Moreover, student models 13405A, 13405B or 13405N) may likewise includesuch additional virtual elements (like one or more of those describedabove with regard to FIGS. 127-133). In other words, additional virtualelements may be presented relative to student models 13405A, 13405B or13405N and teacher model 13408 and such virtual elements may compriseany of the virtual elements disclosed and described elsewhere in thisdisclosure, such as any of those described above (or combinations ofthose described above) with regard to MR educational content 12706 ofFIG. 127, or virtual elements described elsewhere in this disclosure.Also, in some examples, MR/VR student devices 13404 and/or MR/VR teacherdevice 13402 may comprise both a visualization device that presentsvirtual elements to a user and a haptic device that provides touch-basedinformation to the user.

The use of separate virtual models for different students may be highlyuseful for orthopedic surgical education. In some examples, teachermodel 13408 may comprise a virtual model of an anatomical feature, whichmay be registered to an actual bone or soft tissue model (such as anartificial model of bone or actual bone of a cadaver). Similarly, insome examples, student models 13405 may comprise virtual models of ananatomical feature, which may be registered to corresponding physicalmodels or cadaver bones. In other examples, however, teacher model 13408and student models 13405 may be used for education without the need forany physical models or cadavers. If virtual models are being registeredto an actual physical model or a cadaver, then typically MR would beused to combine virtual elements (such as a virtual model) with realworld views that include the actual physical model or cadaver.

A teacher wearing MR/VR teacher device 13402 may manipulate teachermodel 13409 to provide guidance and instruction to the students.Students may then attempt to properly manipulate student models 13405based on the example and tutelage of the teacher. The teacher wearingMR/VR teacher device 13402 is able to view each of student models 13405via MR teacher device 13402. Students wearing MR student devices 13404may be able to view their corresponding student models (e.g., studentdevice 1 13404A may be able to view student model 13405A, and studentdevice 2 13404B may be able to view student model 13405B). In somecases, student devices 13404 may be able to view student modelsassociated with other students, and in other cases, student devices13404 may be unable to view the student models of other students. In anycase, the use of different copies of virtual models for the teacher andthe students can be very useful for orthopedic surgical education byallowing each student to perform their own manipulations on theircorresponding virtual content. The manipulations may include any of thecontrols described herein, such as rotations, re-sizing, repositioning,or other movements of virtual 3D elements. In addition, manipulationsmay include surgical plan selections, such as selections of implants,implant sizes, surgical types, shoulder types, shoulder classifications,or other selections. Also, manipulations may include enabling ordisabling of pre-operative, interoperative or post-operative animations.Any type of virtual control, movement, or virtual selection may beconsidered a manipulation of the virtual content. In some cases, thestudents and/or teacher may compare teacher model 13408 to one or moreof student models 13405 to access whether the student is performing thecorrect steps and correct manipulations of the student content 13405.

MR/VR student devices 13404 may allow students to manipulate and controltheir respective student models 13405 through virtual controls, such asvia gestures, gaze-based controls, voice inputs, combinations of suchvirtual controls, or any control technique useful in mixed reality.Also, manual keypad input, touch screen entry, pointer controls, orother types of controls may be used by students to manipulate respectivestudent models 13405. Similarly, MR/VR teacher device 13402 may allowstudents to manipulate and control teacher model 13408

As noted, teacher model 13408 and student model 13405 may comprisevirtual models of an anatomical feature, such as a virtual model of anexample glenoid bone or an example humeral bone. In addition, teachermodel 13408 and student content 13405 may include additional virtualcontent designed to aid in surgical steps that may be performed on theanatomical feature. For example, each of the content may includesurgical guidance information, registrational content, virtual axes,planes or markers, virtual jigs, virtual guidance on surgical tools,virtual implants, virtual workflow information, virtual range of motioninformation, virtual operative animations, or other virtual information.Using MR/VR student devices 13404, each respective student may beallowed to manipulate their respective student model, e.g., performingsurgical steps, surgical preparation, registration, setting axes fordrilling or reaming, setting a cutting plane, placing a virtual gig,selecting tools, placing virtual implants, choosing virtual workflow,viewing the affects any selection may have on range of motion, and soforth. A teacher wearing MR/VR teaching device 13402 may provide aninstructional example with respect to teacher model 13408 and thestudents wearing MR/VR/student devices 13404 may attempt to mimic thesteps performed by the teacher using their own respective studentcontent (13405A, 13405B or 13405N). Such “hands on” training can be veryhelpful to effectively organize, schedule, and train students. Themodels (as well as outcomes such as shown by virtual range of motioninformation) can be compared to assess the efficacy and effectiveness ofstudent manipulations on the respective student models 13405 relative tothe efficacy and effectiveness of teacher manipulations on the teachermodel 13408.

Student and teacher models shown in FIG. 134 may also be very useful inan educational setting that focuses on surgical planning and surgicaldecisions or selections. Different students may select surgery type(e.g., anatomical vs reverse shoulder arthroplasty) or may select amongdifferent sized surgical tools or different sized surgical implants. Theteacher may provide instructional guidance and tutelage using teachermodel 13402 and student choses and outcomes can be assessed by theteacher by observing and critiquing student models 13405.

FIG. 148 is a conceptual diagram showing a virtual teacher model andmultiple student models, which in some examples may correspond to theteacher and student models illustrated in FIG. 134. Again, although FIG.148 shows an example of shoulder models, similar educational techniquesmay be used for other joints, such as an ankle, in which case the modelsused by teachers and students would be 3D virtual models of ankles orportions thereof. As shown in FIG. 148, using an MR or VR device, ateacher is able to view and manipulate teacher model 14801 throughvirtual controls, such as via gestures, gaze-based controls, voiceinputs, or any control technique useful in mixed reality or virtualreality. Also, manual keypad input, touch screen entry, pointercontrols, combinations of any virtual control mechanisms, or other typesof controls may be used by teacher to manipulate teacher model 14801.Using an MR or VR device, for example, a teacher may rotate, re-size,reposition, or otherwise move teacher model 14801 in space. Moreover, ateacher may show anatomical movement of bones within a shoulder socket.The example teacher model 14801 comprises a 3D model of a shoulder,along with virtual elements showing a virtual 3D representation ofshoulder implant components 14803 and virtual elements showing a likelyimpingement point 14805 that may be caused by shoulder implantationcomponents 14803. The teacher may be able to rotate the virtual humeralbone relative to the glenoid to show shoulder motion. The teacher mayalso be able to enable or disable viewing of different illustratedelements, which may be segmented. Although FIG. 148 shows models in theform of shoulder illustrations with an implant, any type of anatomicalelements, virtual surgical plans, or virtual results may be presented asteacher and student models according to this disclosure in order toachieve education no orthopedic surgery.

Like the teacher's ability to manipulate and control teacher model14801, using an MR or VR device, students can manipulate student models14802A, 14802B and 14802N (collectively student models 14802) viavirtual controls, gestures, gaze-based controls, voice inputs, manualkeypad input, touch screen entry, pointer controls, combination of suchvirtual controls, or any control technique useful in mixed reality orvirtual reality. Like the teacher, for example, students may rotate,re-size, reposition, or otherwise move student models 14802 in space.Moreover, students may enable or disable viewing of different componentsor segmentations of student models 14802, cause shoulder movement,enable or disable viewing of implants 14804A, 14204B and 14204N, presentother implants, enable or disable viewing of impingements 14806A,14806B, and 14806N or otherwise manipulate student models 14802. Again,the models illustrated in FIG. 148 are exemplary of some specificvirtual elements that form teacher model 14801 and student models 14802.This disclosure contemplates a wide variety of 3D models that could beuseful for orthopedic surgical education, including but not limited toany of the 3D models described throughout this disclosure. According toFIGS. 134 and 148, the presentation and use of different models for theteacher and the students can be very useful for educating students onorthopedic surgery, orthopedic surgery planning, and expected results.

FIG. 135 is another conceptual block diagram of an educational system,e.g., educational system 13501 that uses mixed reality and/or virtualreality for orthopedic surgical education. In the example of FIG. 135, ateacher wears MR/VR teacher device 13502 and multiple students wearMR/VR student devices 13504A and 13405B through 13405N (collectivelyMR/VR student devices 13504). In this example, any of the participants(teacher or students) may participate in the teaching session usingmixed reality or virtual reality. That is to say, any of MR/VR teacherdevice 13502 and MR/VR student devices 13504 may comprise mixed realitydevices such as visualization device 213, or alternatively, MR/VRteacher device 13502 and MR/VR student devices 13504 may comprisevirtual reality devices the present only virtual information, in whichcase real-world information presented to users wearing mixed realitydevices may be presented as virtual information to users wearing virtualreality devices. MR/VR educational content 13506 may comprise any of thevirtual content disclosed elsewhere in this disclosure, such as any ofthe educational content described with respect to FIGS. 127-133 orelsewhere in this disclosure. Also, in some cases, MR/VR student devices13504 and/or MR/VR teacher device 13502 may comprise both avisualization device that presents virtual elements to a user and ahaptic device that provides touch-based information to the user.

With the example shown in FIG. 135, MR educational content 13406 maycomprise assignable model 13508. For example, a teacher wearing MR/VRteaching device 13502 may have control over assignable model 13508 andmay be allowed to assign general or specific manipulation rights to oneor more students, so that the one or more students are able to controland manipulate assignable model 13508 when manipulation rights areassigned to them. Thus, in this example, rather than using separateteacher and student models, one assignable model 13508 is presented inthe mixed reality or virtual reality presentation. Using MR/VR teacherdevice 13502, the teacher can manipulate assignable model 13508, andwhen desired, the teacher may assign control of assignable model 13508to a student associated with one of MR/VR student devices 13504. In somecases, a teacher may provide total control rights of assignable model13508, and in other cases, only limited manipulation rights may beassigned to students. For example, in some cases, a teacher may assignrights for a student to select an implant or make some otherpre-operative, interoperative or post-operative selection, but theability to move, re-align, or resize the virtual model may be retainedby the teacher and not assigned to the student. Such limited assignmentof manipulation rights may be useful especially for an educationalsetting that has multiple students viewing the same 3D virtual model.

Educational system 13501 may be useful in demonstrating at least oneaspect of an orthopedic surgical procedure. System 13501 may comprise afirst device (e.g., MR/VR teacher device 13502) configured to display apresentation to a first user (i.e. the teacher) wherein the presentationincludes one or more assignable virtual elements (e.g., assignable model13508). System 13501 may further comprise a second device (e.g., one ofMR/VR student devices 13504) configured to display the presentation to asecond user (i.e., one of the students). According to this example, theone or more assignable virtual elements (e.g., assignable model 13508)demonstrate at least one aspect of the orthopedic surgical procedure,and control of the one or more virtual elements are assignable from thefirst device (e.g., MR/VR teacher device 13502) to the second device(e.g., one of MR/VR student devices 13504). Assignable model 13508, forexample may comprise a 3D virtual representation of one or moreanatomical features, such as a shoulder or a portion of a shoulder orany of the other 3D virtual representations or virtual models describedherein.

Assignable model 13508 may be useful for an educational setting. Ateacher wearing MR/VR teacher device 13502 may manipulate assignablemodel 13508 to provide guidance and instruction to the students. UsingMR/VR teacher device 13502, the teacher may then assign manipulationrights for the assignable virtual model 13508 to one or more studentswearing MR/VR student devices 13504, such as by selecting an assignicon, or using gaze and/or hand gestures to identify the assigneestudent. Once manipulation rights are assigned to one or more students,the student or students may be able to manipulate assignable model13508. The teacher wearing MR/VR teacher device 13502 and each of thestudents wearing MR/VR student devices 13504 may be able to viewassignable model 13508, but only those students that are grantedmanipulation rights (or those granted limited manipulation rights or asubset of manipulation rights) by the teacher are able to control andmanipulate assignable model 13508. Upon assigning manipulation rights toa student, the manipulations and controls by the student may be viewableto the entire class, or possibly viewable only by the teacher device andthe student device of the student that is given manipulation rights.

Assignable control over assignable model 13508 can be very useful fororthopedic surgical education by allowing students to performmanipulations on the assignable model under the guidance of the teacherand the other students. This may amount to a group teaching exercisewhere different students are given a chance to manipulate assignablevirtual model 13508 while being watched by the class.

Similar to other examples described herein, assignable model 13508 maycomprise a virtual model of an anatomical feature, such as a virtualmodel of an example glenoid bone or an example humeral bone, or anentire shoulder. In addition, assignable model 13504 may includeadditional virtual content designed to aid in surgical planning,surgical steps, or post operative analysis that may be performed on theanatomical feature. For example, assignable model 13508 may include anyof the virtual elements described above (or combinations of thosedescribed above) with regard to MR educational content 12706 of FIG.127.

Using MR/VR student devices 13504, once MR/VR teacher device 13502grants control, a student or students may be allowed to manipulateassignable virtual model 13508 in front of the class, e.g., performingsurgical steps, surgical preparation, registration, setting axes fordrilling or reaming, setting a cutting plane, placing a virtual gig,selecting tools, placing virtual implants, choosing virtual workflows,viewing the affects any selection may have on range of motion, and soforth. A teacher wearing MR/VR teaching device 13402 may provideinstructional guidance to effectively train students. The assignablevirtual model 13508 may be reset by MR/VR teacher device 13502 after themanipulation is completed and analyzed for each given student. Outcomes(such as shown by virtual range of motion information) can be comparedamongst the students to assess the efficacy and effectiveness of studentmanipulations on the assignable virtual model 13508 relative to theefficacy and effectiveness of manipulations by other students onassignable virtual model 13508.

In some cases, the act of assigning the assignable model 13508 may beakin to calling on a student in class. The teacher wearing MR/VRteaching device 13502 may grant user controls or editing rights to agiven student, such as by selecting an “assign” widget from a userinterface of MR/VR teacher device 13502, or by using hand gestures,gaze-based selections, or combinations of these selection techniques. Insome cases, a teacher wearing MR/VR teaching device maintains controlover assignable virtual model 13508 and the ability to manipulateassignable virtual model 13508. At the same time, however, a teacherwearing MR/VR teaching device may grant access to one or more students,so that they can perform steps or procedures on assignable virtual model13508 in front of the virtual class.

In some examples, a visualization device 213 configured to assist ateacher in education about an orthopedic surgical procedure ma compriseone or more processors 514 configured to generate one or more virtualelements generate one or more virtual elements wherein the one or morevirtual elements comprise a 3D virtual representation of one or moreanatomical features associated with the orthopedic surgical procedure,and a screen 520, which may comprise a transparent mixed reality displayscreen (such as a see-through holographic lens) configured to presentthe one or more virtual elements as part of a mixed reality presentationby the teacher for one or more students. The mixed reality presentationis configured to promote instruction of the one or more students wearingother visualization devices about the orthopedic surgical procedure,wherein processor 514 is configured to control the one or more virtualelements and wherein processor 514 is further configured to assigncontrol of the one or more virtual elements to one of the other users ofthe other visualization devices. In some cases, processor 514 may assigncontrol to a student in response to input from the teacher. In otherexamples, processor 514 may be configured to assign control based oninput from the one or more students to one of the other visualizationdevices (worn by a student) wherein the other visualization device (wornby the student) communicates the input to visualization device 213 wornby the teacher.

In various examples, the one or more virtual elements generated byprocessor 514 may include any of those described above (or combinationsof those described above) with regard to MR educational content 12706 ofFIG. 127.

In some examples, visualization device 213 may generate one or morevirtual elements comprising a 3D virtual representation of one or moreanatomical features associated with the orthopedic surgical procedure,receive user input to control the one or more virtual elements foreducation about an orthopedic surgical procedure, and receive user inputto assign control of at least some of the virtual elements to anotherdevice. As examples, the virtual representation of the one or moreanatomical features may comprise a 3D virtual model of a human shoulder,or a 3D virtual illustration of a humeral head or a 3D virtualillustration of a glenoid, receiving user input to control the one ormore virtual elements may comprise visualization device 213 beingcontrolled by the teacher and presenting one or more steps of a surgicalplan relative to the 3D virtual representation, and then, afterreceiving the user input, visualization device 213 may assign control ofat least some of the virtual elements to the other device, and receiveadditional input from the other device (e.g. a student device) to adjustone or more steps of the surgical plan relative to the 3D virtualrepresentation

FIG. 136 is a flow diagram illustrating a general educational techniquethat may be performed by a visualization device according to thisdisclosure. The visualization device may comprise visualization device213 described in detail in this disclosure, and may be worn by any typeof student, such as a patient, a surgeon, a physician, a medicalstudent, a technician, a nurse, a physician assistant, relatives of apatient, a researcher, or any other person that may desire educationabout an orthopedic medical procedure.

As shown in FIG. 136, visualization device 213 presents one or morevirtual elements comprising a 3D virtual representation of one or moreanatomical features associated with the orthopedic surgical procedure(13601). For example, a person wearing visualization device 213 may viewa virtual model of an anatomical feature of a patient, such as a virtualshoulder model, a glenoid model, humeral bone model, or anotheranatomical 3D model. Visualization device 213 may then demonstrate atleast one aspect of an orthopedic surgical procedure to the personwearing visualization device 213 (13602). In some cases, the act ofpresenting the one or more virtual elements on visualization device 213(13601) may be educational to the user so as to demonstrate at least oneaspect of the orthopedic surgical procedure (13602). In other cases, thestep of demonstrating at least one aspect of the orthopedic surgicalprocedure on visualization device 213 (13602) may involve some type ofchange or manipulation of the one or more virtual elements onvisualization device 213. In such cases, these changes or manipulationsof the one or more virtual elements on visualization device 213 maycomprise animations, visualizations, or possibly demonstrations by ateacher wearing a different visualization device or a virtual realitydevice. The student wearing visualization device 213 may view suchanimations, visualizations, or demonstrations performed by a teacher,and with the aid of the one or more virtual elements presented onvisualization device 213, the student can be educated on one or moreaspects of the surgical procedure.

In addition to presenting the 3D virtual representation, as examples,the act of presenting one or more virtual elements (13601) may furthercomprise presenting virtual pre-operative planning information,presenting virtual interoperative surgical guidance information, orpresenting post-operative analysis. Visualization device 213 may beconfigured to present one or more virtual elements including any ofthose described above (or combinations of those described above) withregard to MR educational content 12706 of FIG. 127. This virtualinformation presented by visualization device 213 may help todemonstrate at least one aspect of an orthopedic surgical procedure(13602), such as by demonstrating a preoperative plan or portionthereof, demonstrating an interoperative surgical step, or providingpre-operative, inter-operative or post-operative analysis.

In one example, visualization device 213 may present one or more virtualelements (13601) in the form of guidance information that is presentedrelative to an anatomical model or an anatomical feature of a cadaver.In this case, the surgical guidance information may allow the user ofanother visualization device to train the student wearing visualizationdevice 213 on the surgical procedure using a demonstration that isviewed by the student wearing visualization device 213. Example virtualelements that may be useful to present for demonstrating at least oneaspect of a surgical procedure (13601) may comprise a reaming axispositioned relative to a synthetic model glenoid or the glenoid of acadaver.

In yet other examples, visualization device 213 may present one or morevirtual elements (13601) that comprise pre-operative animations, e.g.,which may show desirable results or problems associated with any givenimplantation or implantation procedure. For example, visualizationdevice 213 may present animations showing possible impingementsassociated with the implantation of an incorrectly sized implant oranimations showing a desirable outcome associated with the implantationof a correctly sized implant. In some cases, visualization device 213may present instant feedback to user, which demonstrates or illustratesramifications (e.g., desirable outcomes, or undesirable impingements orloss of range of motion) associated with a particular implant or aparticular implantation procedure.

FIG. 137 is a flow diagram illustrating a general educational techniquethat may be performed by two different devices (e.g., a teacher deviceand a student device) according to this disclosure. The first and seconddevices may each comprise a visualization device 213 described in detailin this disclosure. Alternatively, one or both of the first and seconddevices may comprise virtual reality (VR) devices. As shown in FIG. 137,one or more virtual elements are presented on a first device, whereinthe one or more virtual elements are controllable by a first user (e.g.,a teacher) to demonstrate at least one aspect of an orthopedic surgicalprocedure (13701). The one or more virtual elements are also presentedon a second device to educate a second user (e.g., a student) about theorthopedic surgical procedure (13702). In general, anyone being taughtwith the aid of mixed reality (or virtual reality) may be a studentaccording to this disclosure, and similarly, anyone that teaches withthe aid of mixed reality (or virtual reality) may be a teacher accordingto this disclosure.

As with other examples, in the example process of FIG. 137, the one ormore virtual elements that are controllable by the first user wearingthe first device and viewable by the second user wearing the seconddevice may facilitate education on an orthopedic surgical procedure. Awide variety of such virtual elements are described above, including forexample, a 3D virtual representation of one or more anatomical features,and additional virtual elements to show pre-operative steps,interoperative guidance steps, or surgical results.

In some examples where mixed reality is used, any of the MR studentdevices described herein may comprise a visualization device (such asvisualization device 213) configured to educate a user about anorthopedic surgical procedure. The visualization device 213 may compriseone or more processors 514 configured to generate one or more virtualelements, and a screen 520, which may comprise a transparent mixedreality display screen (such as a see-through holographic lens)configured to present a presentation to a user, wherein the mixedreality presentation includes the one or more virtual elements, whereinthe one or more virtual elements comprise a 3D virtual representation ofone or more anatomical features associated with the orthopedic surgicalprocedure, and wherein the one or more virtual elements further includeone or more virtual pre-operative plan elements shown or illustratedrelative to the 3D virtual representation, one or more virtual surgicalguidance features shown or illustrated relative to the 3D virtualrepresentation, or one or more surgical results virtually illustrated onthe 3D virtual representation such that the mixed reality presentationis configured to educate the user about the orthopedic surgicalprocedure.

Again, as non-limiting examples, the virtual representation of the oneor more anatomical features may comprise a 3D virtual model of a humanshoulder, a 3D virtual illustration of a humeral head, or a 3D virtualillustration of a glenoid. In addition, the one or more virtual elementsmay include any of those described above (or combinations of thosedescribed above) with regard to MR educational content 12706 of FIG.127.

In still other examples, visualization device 213 may be configured todemonstrate a registration process to register the 3D virtualrepresentation with the physical model of the anatomical element or thecorresponding anatomical feature of the cadaver. In some examples, theone or more virtual elements comprise surgical guidance information,include trialing information associated with a prepared implantationlocation for an implant component, illustrate a registration process forregistering a depth aid element, comprise virtual training informationor virtual visual aids on the use of automated tools that include closedloop control, comprise virtual information about range of motion,include a virtual pre-operative animation, include one or more virtualimplant components positioned relative to the 3D virtual representation,or provide other features that can be educational about the orthopedicsurgical procedure.

FIG. 138 is another flow diagram illustrating an educational techniquethat can be performed with the aid of mixed reality and/or virtualreality. FIG. 138 will be described by referring again to educationalsystem 13401 of FIG. 34, which may be useful in demonstrating at leastone aspect of an orthopedic surgical procedure. However, othereducational systems may also implement techniques similar to that shownin FIG. 138. As shown in FIG. 138, a first device (e.g., MR/VR teacherdevice 13402) may display one or more virtual elements comprising ateacher model 13408 (13801). The teacher wearing MR/VR teacher device13402 demonstrates one or more aspects of an orthopedic surgicalprocedure or a surgical plan using the teacher model 13408 (13802).Similarly, a second device (e.g., one of MR/VR student devices 13404)may present one or more virtual elements comprising a student model13405 (13803). The second device may receive input from a student toperform at least one aspect of the surgical procedure or the surgicalplan using the student model 13405 (13804). In this way, a first set ofvirtual elements comprising teacher model 13408 may be manipulated by ateacher to demonstrate at least one aspect of the orthopedic surgicalprocedure or surgical plan, and additional sets of virtual elementscomprising student models 13405A, 13405B or 13405N, which may be similarto teacher model 13408 may be manipulated by different students. In somecases, the student may attempt to mimic surgical steps performed by theteacher, and the use of student-specific virtual elements for eachstudent (similar to the teacher-specific virtual elements used by theteacher) may provide for a very useful teaching system for orthopedicsurgical education. In other cases, students may attempt to select ordefine a surgical plan, select implant size, select surgical type, ormake other surgical plan decisions based on student models 13405, andthe plan and selections made by each student can be compared to that ofthe teacher, or possibly to a plan defined by a computer algorithm.

In some examples, teacher model 13408 may be positioned by teacherdevice 13402 relative to a physical model or an anatomical feature of acadaver, and the student models 13405A, 13405B or 13405N may bepositioned by one of student devices 13404 relative to a secondanatomical model or an anatomical feature of a second cadaver. That isto say, the teacher and the student may manipulate virtual elements thatare positioned relative to different models or different cadavers.Moreover, additional guidance-based virtual elements may be presentedrelative to the student and teacher models to aid in a trial surgicalprocess on the cadavers.

FIG. 139 is another flow diagram illustrating an educational technique,e.g., which may be performed by a visualization device 213, which mayalso be shown as MR/VR teacher device 13502 of FIG. 135. As shown inFIG. 139, visualization device 213 presents one or more virtual elementsto the user (e.g., the teacher), wherein the virtual elements comprise avirtual 3D model of an anatomical feature (13901). Using visualizationdevice 213, the teacher demonstrates at least one aspect of anorthopedic surgical procedure or an orthopedic surgical plan (13902),e.g., by using and manipulating the virtual 3D model and selecting orpresenting other virtual features relative to the virtual 3D model. Theteacher may then assign virtual control of the virtual elements (or asubset of such virtual controls) to one of the students (13903) who useda different visualization device 213 than that worn by the teacher. Oncevirtual control of the assignable virtual elements is granted to thestudent, the visualization device 213 worn by the student may thenreceive student input to perform at least one aspect of the orthopedicsurgical procedure or to define one or more aspects of a surgical planusing the virtual 3D model (13904). In this way, a teacher may usevisualization device 213 to generate and control virtual elements andthen assign control to a student (wherein the student uses anothervisualization device 213). The teacher may observe and assess thestudent's ability to mimic the orthopedic surgical steps performed bythe teacher, or to assess the student's ability to make goodpre-operative decisions and selections for a surgical plan.

FIG. 140 is a conceptual block diagram of an educational system, e.g.,educational system 14001 that use mixed reality and/or virtual realityfor orthopedic surgical education, where a user is able to launch amanipulatable copy of virtual content that includes virtual elements. Inthe example of FIG. 140, a teacher wears MR/VR teacher device 14002 andmultiple students wear MR/VR student devices 14004A and 14004B through14004N (collectively MR/VR student devices 14004). In this example, anyof the participants (teacher or students) may participate in theteaching session using mixed reality or virtual reality. That is to say,any of MR/VR teacher device 14002 and MR/VR student devices 14004 maycomprise mixed reality devices such as visualization device 213, oralternatively, MR/VR teacher device 14002 and MR/VR student devices14004 may comprise virtual reality devices the present only virtualinformation, in which case real-world information presented to userswearing mixed reality devices may be presented as virtual information tousers wearing virtual reality devices.

MR/VR educational content 14006 may comprise any of the virtual contentdisclosed elsewhere in this disclosure, such as any of the educationalcontent described with respect to FIGS. 127-133 or elsewhere in thisdisclosure. In addition, with the example shown in FIG. 140, MReducational content 14006 may include a virtual model 14008, as well asa copy 14009 of the virtual model, which may be launched by one of MR/VRstudent device 14004 (or MR/VR teacher device 14002) during aneducational session. Virtual model 14008 and the copy 14009 of thevirtual model may each comprise a 3D virtual representation of one ormore anatomical features associated with the orthopedic surgicalprocedure, such as a 3D virtual model of a human shoulder, a 3D virtualillustration of a humeral head, a 3D virtual illustration of a glenoid,or another 3D model of anatomical features associated with anyorthopedic surgical procedure. In some cases, MR/VR student devices14004 and/or MR/VR teacher device 14002 may comprise both avisualization device that presents virtual elements to a user and ahaptic device that provides touch-based information to the user.

The ability for students (or the teacher) to launch a copy of a virtualmodel, e.g., virtual model copy 14009, during an educational session maybe very useful for orthopedic surgical procedure education. Virtualmodel copy 14009, for example, may be launched in the middle of aneducational session, after MR/VR teacher device 14002 has been used tomanipulate virtual model 14008. In some cases, virtual model copy 14009may comprise virtual content indicative of an interoperative surgicalstep, where the teacher has already performed some priorsurgical-related manipulations on virtual model 14008. In other cases,virtual model copy 14009 may be launched to allow the student to performsurgical planning steps, selections of implants, selections of surgicaltypes (e.g., reverse vs. anatomic) placement of implants, selections ofimplant sizes, selection of surgical tools, or other surgical planningsteps or decisions.

Educational system 14001 may be useful in demonstrating at least oneaspect of an orthopedic surgical procedure, such as surgical steps or asurgical plan. A teacher wearing MR/VR teacher device 14002 maydemonstrate one or more aspects of a surgical steps or the surgicalplanning. During the demonstration or at any time, one of MR/VR studentdevice 14004 (or MR/VR teacher device 14002) may launch virtual modelcopy 14009, which is a copy of virtual model 14008, possibly after theteacher has performed one or more surgical-related manipulations onvirtual model 14008 or some prior surgical planning steps on virtualmodel. This way, a student may be able to focus on a particular planningstage or a particular interoperative surgical step of the procedure. Insome cases, virtual model copy 14009 may be discarded after studentmanipulations and teacher review, but in other cases, virtual model copy14009 may be adopted by the teacher so as to replace virtual model14008. In this later example, student manipulations of virtual modelcopy 14009 may be accepted by the teacher and adopted as virtual model14008 as the teacher continues to educate the students on the next stepof the orthopedic surgical procedure or the next step in the surgicalplanning.

In some examples, system 14001 may comprise a first device (e.g., MR/VRteacher device 14002) configured to display a presentation to a firstuser (i.e., a teacher) wherein the presentation includes virtual model14008 comprising one or more virtual elements that are controllable bythe first user wearing the first device and wherein the one or morevirtual elements comprise a 3D virtual representation of one or moreanatomical features associated with the orthopedic surgical procedure.Moreover, system 14001 may comprise a second device (e.g., one of MR/VRstudent devices 14004) configured to display the presentation to asecond user (i.e., one of the students). Virtual model 14008 comprisingone or more virtual element may demonstrate at least one aspect of theorthopedic surgical procedure, such as a surgical step of the procedureor a surgical planning step for the procedure. The second device (e.g.,one of MR/VR student devices 14004) or the first device (e.g., MR/VRteacher device 14002) may be configured to generate a copy 14009 of thevirtual model in response to input from the second user (i.e., one ofthe students) or the first user (i.e., the teacher). And the copy 14009of the virtual model may be controllable by the second user wearing thesecond device. In some cases, the first device may be further configuredto replace virtual model 14008 with the copy 14009 of the virtual modelin the presentation, after the copy 14009 of the virtual model ismanipulated by the second user wearing the second device. In otherwords, for example, if the student performs the correct steps ormanipulations on the student's copy of the virtual content, the teachermay adopt the student's copied version of the virtual model after suchstudent manipulations as the new teacher content for all students, andthe teacher may then continue the educational session to demonstratelater surgical steps or surgical planning steps of the orthopedicsurgical procedure.

As examples, virtual model 14008 (as well as virtual model copy 14009)may comprise additional virtual elements that illustrate pre-operativeplan elements relative to the 3D virtual representation, one or morevirtual surgical guidance features illustrated relative to the 3Dvirtual representation, or one or more surgical results virtuallyillustrated on the 3D virtual representation. A wide variety of suchadditional virtual elements are described throughout this disclosure.

In some examples, a visualization device 213 configured to educate auser about an orthopedic surgical procedure may comprise a screen 520,which may comprise a transparent mixed reality display screen (such as asee-through holographic lens) configured to present a mixed realitypresentation to a user, wherein the mixed reality presentation includesthe one or more virtual elements comprising a 3D virtual representationof one or more anatomical features associated with the orthopedicsurgical procedure. In other words, a visualization device 213 worn by astudent may view a mixed reality presentation on orthopedic surgicalprocedures (e.g. surgical steps or surgical planning) that is generatedand controlled by a different visualization device worn by a teacher.The visualization device 213 worn by the student (or that worn by theteacher) may comprise a processor 514 configured to generate a copy of avirtual model shown by the teacher, wherein the copy is controllable byvisualization device 213 worn by the student. In this way, a student (orthe teacher) may be able to launch and manipulate a copy of virtualcontent presented by the teacher during a presentation by the teacher.The copy of virtual content that is launched and manipulated by thestudent may include surgical planning (or surgical steps) that theteacher may have already performed with regard to the virtual contentbefore such content is launched and manipulated by the student. In thisway, student copies of virtual models may be designed for teachingspecific surgical planning steps or specific surgical operation steps.

As examples, the virtual model that may be launched and manipulated by astudent as a copy of teacher model may comprise any of the additionalvirtual information described herein in order to facilitate education onsurgical planning steps or surgical operating steps. Such additionalvirtual information, for example, may comprise such things as a virtualcutting axis shown or illustrated relative to the 3D virtualrepresentation, a virtual reaming axis shown or illustrated relative tothe 3D virtual representation, a virtual drilling axis shown orillustrated relative to the 3D virtual representation, placement orselection of a virtual jig or guide relative to 3D virtualrepresentation, a virtual axis shown or illustrated relative to avirtual jig or guide, surgical guidance information presented relativeto the 3D virtual representation, an illustration of a registrationprocess for registering the 3D virtual representation to a physicalmodel or a corresponding feature of a cadaver, trialing informationassociated with a prepared implantation location for implant component,information showing a registration process for registering a depth aidelement, virtual training information or virtual visual aids on the useof automated tools that include closed loop control, virtual informationabout range of motion, a virtual pre-operative animation, anillustration of one or more virtual implant components relative to the3D virtual representation, selections of decisions of surgical plan,selections of implants or implant sizes, selection of surgical tools ortool sizes, selections or decisions of surgical type, or other elementsrelated to surgical planning or surgical guidance.

In some examples, a method may comprise displaying a mixed realitypresentation of an orthopedic surgical procedure on a student device anda teacher device, wherein the mixed reality presentation includes ateacher copy of virtual elements controlled by a teacher device andwherein the virtual elements comprise a 3D virtual representation of oneor more anatomical features associated with the orthopedic surgicalprocedure; and

In some cases, MR/VR teacher device 14002 may always include a mastermodel of virtual model 14008. One or more copies (e.g., virtual modelcopy 14009) may be generated for student use by one of MR/VR studentdevices 14004, and in some cases, such copies (e.g., virtual model copy14009) may be presented as a comparison to a master model (e.g., virtualmodel 14008) controlled by MR/VR teacher device 14002. In some cases,virtual model copy 14009 may be superimposed on virtual model 14008,possibly after student manipulation, to allow for precise comparisonbetween virtual model 14008 and virtual model copy 14009. In some cases,each student may have corresponding virtual model copy 14009. In somecases, the virtual model copy 14009 for each student may be viewableonly by that corresponding MR/VR student devices 14004, and in othercases each of MR/VR student devices may be able to view virtual modelcopy 14009 associated with other ones of MR/VR student devices 14004.

FIG. 141 is a conceptual block diagram of an educational system, e.g.,educational system 14101 that use mixed reality and/or virtual realityfor orthopedic surgical education where students and teachers are ableto view and compare several different 3D virtual models, which mayinclude additional virtual elements as described here. In the example ofFIG. 141, a teacher wears MR/VR teacher device 14102 and multiplestudents wear MR/VR student devices 14104A and 14104B through 14104N(collectively MR/VR student devices 14104). In this example, any of theparticipants (teacher or students) may participate in the teachingsession using mixed reality or virtual reality. That is to say, any ofMR/VR teacher device 14102 and MR/VR student devices 14104 may comprisemixed reality devices such as visualization device 213, oralternatively, MR/VR teacher device 14102 and MR/VR student devices14104 may comprise virtual reality devices the present only virtualinformation, in which case real-world information presented to userswearing mixed reality devices may be presented as virtual information tousers wearing virtual reality devices. MR/VR educational content 14106may comprise any of the virtual content disclosed elsewhere in thisdisclosure, such as any of the educational content described withrespect to FIGS. 127-133 or elsewhere in this disclosure. Also, in somecases, MR/VR student devices 14104 and/or MR/VR teacher device 14102 maycomprise both a visualization device that presents virtual elements to auser and a haptic device that provides touch-based information to theuser.

As shown in the example of FIG. 141, MR/VR educational content 14106includes a plurality of models (e.g., model 1 14108A and model 2 14108Bthrough model N 14108N). In some cases, the plurality of models(collectively models 14108) may show different surgical plans on thesame anatomy. Presenting a plurality of models 14108 as part of acollective presentation may be very useful to promote education of anorthopedic surgical procedure or surgical planning. This may present aside-by side comparison of a current surgical plan relative to otherexamples. For example, model 1 14108A may comprise a current patientmodel planned for surgery, and the other models (model 2 14108B throughmodel N 14108N) may comprise models planned by others based on the sameanatomy, models associated with achieved case studies, models oftheoretical cases, models generated by a computer algorithm, or any casestudy that may be useful to compare with model 1 14108B. In otherexamples, the plurality of models 14108 may comprise student modelsrelative to a teacher model, e.g., which may be presented and comparedat different stages of an educational orthopedic surgical process, orpresented to show different surgical plans different aspects of thesurgical plan. In some cases, MR/VR student devices 14104 may be worn bysurgeons that collaborate and share experiences with one another bypresenting different models in a side-by-side comparison, within an MRor VR presentation. In some cases, MR/VR student devices 14104 mayparticipate in a “crowd sourcing” session in which the differentstudents collaborate on a similar problem or issue with the use of MR orVR to showcase presentation of a case study of a specific patient, or aspecific issue in a particular case study.

In some cases, the plurality of models 14108 may be selected from acatalog of 3D virtual representations (e.g., stored in memory of MR/VRstudent device 14104 or MR teacher device 14102 or stored remotely). Thedifferent models in the catalog may demonstrate a wide variety ofdifferent shoulder conditions that may require orthopedic surgicalrepair. A teacher using MR/VR teacher device 14102, for example, mayselect one or more 3D virtual representations from the catalog of 3Dimages in order to make educational demonstrations to a student wearingone of MR/VR student devices 14104. In some cases, 3D virtual models ofpatient anatomy may be compared to 3D virtual models in the catalog forside-by-side comparisons, which can help users identify similaritiesbetween the patient anatomy and other example antinomy from the catalogthat illustrates one or more shoulder problems.

In still other cases, the plurality of models 14108 may be selected toillustrate shoulders with different types of classification (e.g.different types of Walch classifications). The different classificationsmay call for different types of surgical procedures or selection ofdifferent implant components with particular sizes, angles, and implantpositions. Preoperative, interoperative or post-operative steps may bedemonstrated side-by-side in the plurality of models 14108 in order toillustrate differences in procedures desirable for different types ofshoulder classifications. In still other cases, the plurality of models14108 may comprise ankle models used for demonstrating one or moreaspects of an orthopedic ankle procedure.

In some cases, different ones of models 14108 may illustrate differentsurgical procedures or different surgical plans associated with a commonpatient, e.g., to help the users of MR teacher device 14102 and MRstudent devices 14104 identify a most desirable surgical procedure orplan for a given shoulder classification. For example, model 1 14108Amay present a model of an anatomical surgical implant and model 2 14108Bmay present a model of a reverse surgical implant. In this way, MRteacher device 14102 and MR student devices 14104 may compare implants,possibly for pre-operative determinations to identify the best type ofsurgical procedure for a given shoulder classification. In someexamples, MR teacher device 14102 and MR student devices 14104 may beworn by a panel of experts or a panel of instructors that exchange orshare the teaching roll. In some cases, each user may have the abilityto take control or be granted control to become MR teacher device 14102.

FIG. 142 is a conceptual block diagram of an educational system, e.g.,educational system 14201 that use mixed reality and/or virtual realityfor orthopedic surgical education where a teacher has a teacher controlmenu that comprises virtual control elements to allow the teacher tocontrol MR/VR educational content. Like other examples, in the exampleof FIG. 142, a teacher wears MR/VR teacher device 14202 and multiplestudents wear MR/VR student devices 14204A and 14204B through 14204N(collectively MR/VR student devices 14204). In this example, any of theparticipants (teacher or students) may participate in the teachingsession using mixed reality or virtual reality. That is to say, any ofMR/VR teacher device 14202 and MR/VR student devices 14204 may comprisemixed reality devices such as visualization device 213, oralternatively, MR/VR teacher device 14202 and MR/VR student devices14204 may comprise virtual reality devices the present only virtualinformation, in which case real-world information presented to userswearing mixed reality devices may be presented as virtual information tousers wearing virtual reality devices. MR/VR educational content 14206may comprise any of the virtual content disclosed elsewhere in thisdisclosure, such as any of the educational content described withrespect to FIGS. 127-133 or elsewhere in this disclosure. Also, in somecases, MR/VR student devices 14204 and/or MR/VR teacher device 14202 maycomprise both a visualization device that presents virtual elements to auser and a haptic device that provides touch-based information to theuser.

As shown in the example of FIG. 142, MR/VR educational content 14206comprises a teacher control menu 14205, which may comprise virtualcontrol elements for controlling an MR or VR presentation on anorthopedic surgical procedure. In some cases, teacher control menu 14205may be visible only to the teacher wearing MR/VR teacher device 14202and may be hidden from the MR/VR educational content that is shown onMR/VR student devices 14204.

Teacher control menu 14205 may comprise selectable elements, such asvirtual control widgets, that can be selected by a teacher wearing MR/VRteacher device 14202 in order to control an MR or VR presentation usinggazes, gestures towards the widgets, voice commands or other ways ofselection. In particular, teacher control menu 14205 may presentelements for controlling or launching MR or VR features or elementsdescribed in this disclosure, and users may select such elements usinggazes, gestures, voice controls or other selections used in mixedreality or virtual reality. For example, teacher control menu 14205 maycomprise one or more elements for presenting and/or manipulating MR/VReducational content 14206, which may comprise any of the virtual contentdescribed herein for use in orthopedic surgical procedures or orthopedicsurgical procedure education. Teacher control menu 14205 may compriseeducational tools such as icons for launching videos, images,presentations (such as power point presentations), spreadsheets, oranything that may be useful in a mixed reality or virtual realityteaching environment.

As one example, teacher control menu 14205 may comprise a virtualelement or icon for assigning virtual control of MR/VR educationalcontent 14204 to one of MR/VR student devices 14204. In another example,teacher control menu 14205 may comprise a virtual element or icon forlaunching a student copy of virtual content for a specific MR/VR studentdevice, or for launching multiple copies of virtual content for all ofthe MR/VR student devices 14204. More generally, teacher control menu14205 may include one or more virtual elements or icons for presentingteacher content that is manipulatable by the teacher wearing MR/VRteacher device 14202 or for presenting student content that ismanipulatable by students wearing MR/VR student devices 14204. Thevirtual elements or icons may be selected via gazes, gestures, voicecontrols or other selections used in mixed reality or virtual reality

In other examples, teacher control menu 14205 may comprise one or morevirtual elements or icons for presenting multiple models forcollaborative and comparative viewing. In other examples, teachercontrol menu 14205 may comprise one or more virtual elements or iconsfor presenting video, for presenting CT scans, images or segmentations,for presenting pre-operative images or videos, for selecting orlaunching presentations or presentation tools (such as a power pointpresentation). In other examples, teacher control menu 14205 may includea notepad for notetaking, a record icon for recording an educationalsession or recording a portion of an educational session. In otherexamples, teacher control menu 14205 may comprise virtual elements oricons for archiving content of a given educational session, foraccessing recorded videos of other cases, or for loading prior examplesor prior models for class demonstrations or discussion.

In still other examples, teacher control menu 14205 may comprise one ormore virtual elements or icons for sharing the ability to view virtualcontent or for sharing the ability to control such virtual content. Thesharing elements or icons may allow for sharing with specific ones ofMR/VR student devices 14204 or for sharing with all MR/VR studentdevices 14204.

In other examples, teacher control menu 14205 may comprise a file queue,e.g., for organizing a class session. Also, teacher control menu 14205may include content related to educational credit (such as continuingmedical education credit), such as for presenting the availability ofsuch credit or for soliciting student responses to verify studentattendance.

In some examples, teacher control menu 14205 may include selectableelements or icons for assigning editing rights to some or all virtualelements. In some examples, teacher control menu 14205 may comprise adrop-down menu of participants, facilitating the ability of MR/VRteacher device 14002 to select or call on other participants wearingMR/VR student devices 14204. In other examples, teacher control menu14205 may comprise selectable icons or avatars that can be selected bythe user of MR/VR teacher device 14202 to call on students or assignvirtual control to students, e.g., using hand gesture or gaze-based usercontrols.

Teacher control menu 14205 may include elements or icons for filtersthat identify similar cases by procedure similarity, e.g., locating oneor more procedures or procedure steps that may be similar to a procedureor step of interest. Upon selection of a specific filter icon by theteacher, MR/VR teacher device 14202 may access a database (locatedlocally or remotely) and identify one or more example case studies orprior surgeries that are close matches to a current case study. MR/VRteacher device 14202 may then load archived models or examples intoMR/VR educational content 14206 for presentation to the students. Theability to identify archived case studies or example procedures thathave similarities with a current case study may be very helpful foreducating and may have particular use in a collaborative educationalsession where students work together (e.g., in a crowd sourcing session)to brainstorm on how best to address a current case study.

The elements or icons presented on teacher control menu 14205 may beselectable by the teacher wearing MR/VR teacher device 14202 via handgestures, gaze-based controls, facial expressions, or possibly with theuse of a selection tool (such as a wand or laser pointer). The teachermay select icons or elements from teacher control menu 14205 to controlthe MR/VR presentation and to control and manipulate virtual content,which may include any of the virtual content described herein for use inorthopedic surgery or orthopedic surgical education, including but notlimited to a 3D virtual representation (e.g., a virtual model of a humanshoulder, a virtual model of a glenoid or glenoid surface, or a virtualmodel of a humeral bone or humeral head), and/or additional virtualcontent such as any of the virtual elements (or combinations) describedwith regard to MR educational content 12706 of FIG. 127.

In still other examples, teacher control menu 14205 may comprise a setof saved surgical plans or illustrations for selecting and demonstratingdifferent procedures such as a reverse shoulder arthroplasty vs ananatomical shoulder arthroplasty.

FIG. 143 is another conceptual block diagram of an educational system,e.g., educational system 14301 that use mixed reality and/or virtualreality for orthopedic surgical education where both a teacher and thestudents have virtual control elements for controlling MR/VR educationalcontent. Like other examples, in the example of FIG. 143, a teacherwears MR/VR teacher device 14302 and multiple students wear MR/VRstudent devices 14304A and 14304B through 14304N (collectively MR/VRstudent devices 14304). In this example, any of the participants(teacher or students) may participate in the teaching session usingmixed reality or virtual reality. That is to say, any of MR/VR teacherdevice 14302 and MR/VR student devices 14304 may comprise mixed realitydevices such as visualization device 213, or alternatively, MR/VRteacher device 14302 and MR/VR student devices 14304 may comprisevirtual reality devices the present only virtual information, in whichcase real-world information presented to users wearing mixed realitydevices may be presented as virtual information to users wearing virtualreality devices. MR/VR educational content 14306 may comprise any of thevirtual content disclosed elsewhere in this disclosure, such as any ofthe educational content described with respect to FIGS. 127-133 orelsewhere in this disclosure. Also, in some cases, MR/VR student devices14304 and/or MR/VR teacher device 14302 may comprise both avisualization device that presents virtual elements to a user and ahaptic device that provides touch-based information to the user.

Teacher control elements 14305 may comprise any of the features,elements, icons, or controls described above with regard to teachercontrol elements 14205 of system 14202 shown in FIG. 143. In the exampleof FIG. 143, however, MR/VR student devices 14304 are also presentedwith virtual controls as part of MR/VR educational content 14206. Forexample, MR/VR student device 1 14304A may be configured to presentcorresponding student 1 control elements 14307 and MR/VR student device2 14304B may be configured to present corresponding student 2 controlelements 14307B. Each of a plurality of student devices havecorresponding virtual controls, and the virtual controls of each devicemay only view viewable by the user of that device. That is to say,student 1 control elements 14307A may be viewable by MR/VR studentdevice 1 14304A, but student 1 control elements 14307A may be unviewableby MR/VR student device 2 14304B. Similarly, student 2 control elements14307B may be viewable by MR/VR student device 2 14304B, but student 2control elements 14307B may be unviewable by MR/VR student device 114304A. In some cases, MR/VR teacher device 14302 may be able to viewonly teacher control elements 14305 and may be unable to view studentcontrol elements 14307, but in other cases, MR/VR teacher device 14302may be able to view all of the control elements including studentcontrol elements 14307 associated with the student devices 14304. Insome cases, teacher control elements 14305 may include icons or elementsfor enabling or disabling the viewability of student control elements14307 by MR/VR teacher device 14302, which may be useful when a teacherneeds to explain to a student how to use such controls.

In general, each of student control elements 14307 may comprise any ofthe features, elements, icons, or controls that are included withinteacher control elements 14305. Moreover, each of student controlelements 14307 may comprise any of the features, elements, icons, orcontrols that are described above with regard to teacher controlelements 14205 of system 14202 shown in FIG. 143. In most cases,however, student control elements 14307 may comprise a more limitednumber of control elements relative to teacher control elements 14305.In most cases, for example, teacher control elements 14305 may provideuniversal control over MR/VR educational content 14206, whereas studentcontrol elements 14307 may have a more limited control over MR/VReducational content 14206. In still other cases, different ones of MR/VRstudent devices 14304 may be afforded different levels of control overMR/VR educational content, and in some cases, MR/VR teacher device 14302may be configured to assign or unassign such different levels of controlto MR/VR student devices 14304.

In some examples, teacher control elements 14305 and/or student controlelements 14307 may include note taking features for recording notes ofthe different users. Also, recordings of the training session and anyvirtual manipulations that are performed by the students or the teachermay be recorded and documented as part of MR/VR educational content14306.

FIG. 144 is a flow diagram illustrating another educational techniquethat can be performed with the aid of mixed reality and/or virtualreality. FIG. 144 will be described from the perspective of system 14001of FIG. 140, although other systems could use a similar technique. Asshown in FIG. 144, an MR/VR device (such as MR/VR teacher device 14002or one of MR/VR student devices 14004) displays an MR or VR presentationon an orthopedic surgical procedure including virtual elements (e.g.,virtual model 14008) that are controlled by MR/VR teacher device 14002(14401). The MR/VR device (such as MR/VR teacher device 14002 or one ofMR/VR student devices 14004) then generates a student copy of thevirtual elements (e.g., virtual model copy 14009) wherein the studentcopy is controllable by a student device (e.g., one of MR/VR studentdevices 14004) (14402). In some cases, MR/VR teacher device 14002generates the student copy for each student, and in some cases, theMR/VR student devices 14004 are able to generate respective studentcopies of the virtual elements. In some cases, the copy is not anoriginal version of the virtual elements, but rather the copy maycomprise a version of the virtual elements after some initialmanipulation by the teacher. For example, the teacher or students may beable to generate copies of the virtual content during an ongoing virtualpresentation, allowing student copies to be generated at differentstages of an orthopedic surgical procedure or different stages of asurgical planning session so that the student can mimic or practicespecific surgical steps or make surgical planning decisions associatedwith specific surgical planning steps. Moreover, in some cases, MR/VRteacher device 14002 may replace virtual model 14008 with the copy ofvirtual model 14009, after the student has manipulated the copy 14009.

In some examples, system 14001 demonstrates at least one aspect of anorthopedic surgical procedure and comprises a first device (e.g., MR/VRteacher device 14002) configured to display a presentation to a firstuser wherein the presentation includes one or more virtual elements thatare controllable by the first user wearing the first device wherein theone or more virtual elements comprise a 3D virtual representation of oneor more anatomical features associated with the orthopedic surgicalprocedure. In addition, system 14001 comprises a second device (e.g.,one of MR/VR student devices 14004) configured to display thepresentation to a second user. The one or more virtual elementsdemonstrate at least one aspect of the orthopedic surgical procedure. Insome cases, the second device (e.g., one of MR/VR student devices 14004)or the first device (e.g., MR/VR teacher device 14002) may be configuredto generate a copy of the one or more virtual elements in response toinput from the first user or the second user and wherein the copy of theone or more virtual elements are controllable by the second user wearingthe second device. In other cases, however, MR/VR teacher device 14002may generate and assign copies to the MR/VR student devices 14004A. Insome examples, the first device may be further configured to replace thecopy of the one or more virtual elements with the copy of the one ormore virtual elements in the presentation after the copy of the one ormore virtual elements is manipulated by the second user wearing thesecond device.

As examples, the one or more virtual elements represented as virtualmodel 14008 (as well as the student copy or copies of the virtualelements represented as virtual model copy 14009) may comprise 3Dvirtual representation of one or more anatomical features as well asadditional surgical guidance information or surgical planninginformation for the 3D virtual representation.

In some examples, a visualization device 213 may be configured toeducate a user about an orthopedic surgical procedure and visualizationdevice 213 may comprise a screen 520, which may comprise a transparentmixed reality display screen (such as a see-through holographic lens)configured to present a mixed reality presentation to a user, whereinthe mixed reality presentation includes the one or more virtual elementscomprising a 3D virtual representation of one or more anatomicalfeatures associated with the orthopedic surgical procedure. In addition,visualization device 213 may further comprise one or more processors 514configured to generate a copy of the one or more virtual elements,wherein the copy is controllable with the visualization device. In somecases, the original mixed reality presentation is generated andcontrolled by another user of another visualization device (e.g., ateacher device) and the copy is generated and controlled byvisualization device 213 worn by a student. In other cases, however,copies may be generated by MR/VR teacher device 14002 and assigned toone of MR/VR student devices 14004. In some examples, a method maycomprise displaying a mixed reality presentation of an orthopedicsurgical procedure on a student device and a teacher device, wherein themixed reality presentation includes a teacher copy of virtual elementscontrolled by a teacher device and wherein the virtual elements comprisea 3D virtual representation of one or more anatomical featuresassociated with the orthopedic surgical procedure, and generating astudent copy of the virtual elements, wherein the student copy iscontrollable by the student device.

FIG. 145 is a conceptual block diagram of an educational system thatincludes features to help educate a remote user on specific details ofan ongoing surgical procedure. Educational system 14501 sets forth aspecific educational scenario of orthopedic surgical education speciallyrelated to an interoperative setting. Other sections of this disclosuredescribe a variety of scenarios and settings where interoperativeinteraction is performed between or amongst different users, and in manycollaborative situations, one or more surgical participants may beremote participants that interact with the operating room participantsthought the use of MR or possibly VR. In some cases, surgical help maybe located in the operating room and in other cases, surgical help maybe located remotely. Regardless, FIG. 145 illustrates an educationalscenario where a surgical participant (such as a surgeon) wearing MRsurgical device 14504 elicits surgical help from another person (such ass surgical expert) wearing or otherwise using MR/VR surgical help device14508.

MR surgical device 14504 may comprise a mixed reality device such asvisualization device 213 described in detail throughout this disclosure.For example, MR surgical device 14504 may comprise a visualizationdevice worn by a surgeon to provide virtual interoperative assistance tothe surgeon. MR surgical device 14504 may implement any of the featuresdescribed in this disclosure to assist a user (e.g., a surgeon orpossibly another surgical participant) with orthopedic surgical steps.

VR/MR surgical help device 14508 may comprise a mixed reality devicesuch as visualization device 213, or alternatively, VR/MR surgical helpdevice 14508 may comprise a virtual reality device. In any case, theuser of MR surgical help device 14504 may elicit expert help during aprocedure, and to do so, MR surgical device 14504 may be configured toinitiate a communication session with VR/MR surgical help device 14508.Or in other examples, expert help may be contacted in another way suchas via a telephone call, e-mail, text message, or any other type ofmessage, and in response to being contacted, the surgical expert mayjoin the procedure occurring in the operating room (either physically orremotely) via VR/MR surgical help device 14508. In some cases, MRsurgical device 14504 and VR/MR surgical help device 14508 maycommunicate with one another directly or via a network.

A surgeon wearing MR surgical device 14504 may benefit from expert helpon a surgical procedure for a wide variety of situations. Anytime thesurgeon encounters difficulties, complications, or unexpectedsituations, surgeon wearing MR surgical device 14504 may benefit fromexpert help. For example, upon opening a shoulder or ankle andattempting to install an implant, a surgeon wearing MR surgical device14504 may discover that the patient's bone quality is too poor for theimplant. In this case, the surgeon wearing MR surgical device 14504 maydetermine that its difficult or impossible to implement a plannedfixation, and in this case, it may be beneficial to contact an expert toshow the expert the patient's CT-images and a 3D reconstruction of thepatient's anatomy, along with the real scene images (e.g., obtained viaMR surgical device 14504). In these and other cases, it may be verybeneficial for a surgical expert wearing VR/MR surgical help device14508 to join the surgical procedure.

Upon joining the surgical procedure, however, the surgical expertwearing VR/MR surgical help device 14508 may benefit from educationalfeatures illustrated as “bring up to speed” features 14505 which mayinform the user of VR/MR surgical help device 14508 on one or morepreviously-executed steps of the orthopedic surgical procedure. Forexample, in order to provide useful help to the user of MR surgicaldevice 14504 at the time a user of VR/MR surgical help device 14508 isengaged, the user of VR/MR surgical help device 14508 may need to beeducated and essentially “brought up to speed” with regard to previoussteps that already occurred in the procedure. Indeed, in order toclassify problems or issues in the ongoing surgical procedure, the userof VR/MR surgical help device may require some knowledge about steps ofthe procedure that were previously performed. Accordingly, content fororthopedic surgical help 14506 may comprise “bring up to speed” features14505 to provide specific interoperative education in a quick andefficient manner. Moreover, in some cases, visual help 14507 may also beprovided, e.g., as visual aids to the user of VR/MR surgical help device14508.

In one example, a surgical system 14501 is configured to provide forinteroperative education to VR/MR surgical help device 14508 during anongoing surgical procedure. System 14501 comprises first device (e.g.,MR surgical device 14504) configured to display a first presentation toa first user (e.g., a surgeon or other surgical participant). The firstpresentation includes one or more virtual elements configured to assistthe first user in an orthopedic surgical procedure. System 14501 mayfurther comprise a second device (e.g., VR/MR surgical help device14508) configured to display a second presentation to a second user(e.g., an expert surgeon or other expert associated with one or moresteps of the orthopedic surgical process). The second user may becontacted during the procedure in order to provide surgical assistanceon the orthopedic surgical procedure. According to this disclosure, inorder to adequately educate the user of the second device, the seconddevice is further configured to display content that informs or educatesthe second user on one or more previously-executed steps of theorthopedic surgical procedure.

In some cases, the first device (e.g., MR surgical device 14504) isconfigured to elicit the surgical assistance from the second user of thesecond device (e.g., VR/MR surgical help device 14508). However, thesurgical assistance could also be elicited in other ways, such as via atelephone call, e-mail, text, or any other type of message. As examples,“bring up to speed” features 14505 may comprise one or more of: scans ofa patient or segmentations of image data of the patient, informationassociated with one or more previously-executed steps of the orthopedicsurgical procedure, information associated with previously-used surgicaltools, information associated with an implanted device, informationassociated with a jig or guide used in the orthopedic surgicalprocedure, information associated with a pre-operative plan, informationassociated with timing of one or more of the previously-executed stepsof the orthopedic surgical procedure, or patient data associated withone or more of the previously-executed steps of the orthopedic surgicalprocedure. In some cases, “bring up to speed” features 14505 may includepatient data that was acquired before the surgery, such as CT-Scans, 3Dreconstructions of the patient anatomy, surgery plans or planninginformation, patient age, patient weight, patient vital information, orany other patient data. In some cases, MR surgical device 14504 mayautomatically track steps of the surgery and in this case, “bring up tospeed” features may include information that identifies the current stepand previously executed steps of the surgical procedure.

Moreover, as noted, visual help 14507 may also help with interoperativeeducation and may be especially useful to accelerate the educationalprocess, which may be important during an ongoing surgical procedure.Accordingly, educational content for orthopedic surgery help 14506 maycomprise visual help 14507, such as one or more images associated withone or more of the previously-executed surgical steps, one or morevideos of one or more of the previously-executed surgical steps, one ormore images or videos of a prepared anatomical surface of a patient thatwas prepared for an implant, animations or illustrations of plannedsurgical steps, one or more images or videos of a surgical plan, and oneor more images or videos of the orthopedic surgical procedure.

In some examples, “bring up to speed” features 14505 and/or visual help147 may comprise one or more images or videos of the orthopedic surgicalprocedure, which may be captured by MR surgical device during theorthopedic surgical procedure or captured by another camera or anotherMR device located in the operating room during the orthopedic surgicalprocedure. In order to facilitate quick and efficient review by the userof VR/MR surgical help device 14508, the images or videos may betime-indexed. Or in some cases, the images or videos may be indexedbased on surgical steps or stages so that VR/MR surgical help device14508 can select particular surgical steps or stages and view images orvideos recorded at those steps or stages. The steps or stages manyinclude pre-operative planning steps, which are typically shown withimages, and inter-operative steps, which may be shown with video orimages.

In some examples, content 14506 may be acquired by cameras located inthe operating room or via one or more cameras on MR surgical device14504 so that the VR/MR surgical help device 14508 can see theperspective of the surgeon (or some other person in the room such as anurse or technician) or from a camera in the room. Such content may alsoinclude any virtual imagery that was presented to the surgeon during theprocedure, and in some examples, VR/MR surgical help device 14508 couldenable or disable the virtual imagery to view patient shoulder anatomyor patient shoulder anatomy plus virtual imagery. Also, a virtualanatomical model could be presented to show the virtual surgical plan,and VR/MR surgical help device 14508 may be able to selectively see thevirtual surgical plan, bone model only, bone model plus virtual guidance(e.g., virtual elements to aid in reaming, cutting, drilling, screwing,or other positioning makers). In still other cases, VR/MR surgical helpdevice 14508 may be able to selectively view a virtual surgical plan,plus a bone model, plus virtual guidance features. Such content may berecorded during the procedure or planning stages and may be stored asimages are video could be time indexed or task indexed (e.g., indexed toa point in the workflow such as glenoid reaming, glenoid implantplacement). Such content may also identify such things as a plate for ananatomical procedure or hemi for reverse anatomical procedure, humeralcutting, humeral implant placement (e.g., hemi with stem or stemless foranatomical or plate with stem or stemless for reverse), and/or otherthings.

The “bring up to speed” features 14505 and/or visual help 147 maycomprise features for informing or educating a remote surgeon or anotherinteroperative surgical participant wearing VR/MR surgical help device14508. Moreover, in some cases, educational content 14506 may alsoinclude features that can aid the wearer of MR/VR surgical help device14508 for educating the person wearing MR surgical device 14504. Inother words, after being “brought up to speed” and sufficiently educatedon the previous steps of the surgical procedure, VR/MR surgical helpdevice 14508 may present features that can aid in providing the experthelp to the user of MR surgical device 14504. For example, educationalcontent 14506 may include an archive of video clips (such as video clipsof the surgery or video clips of similar surgeries) that can be accessedand presented by MR/VR surgical help device 14508 to MR surgical device14504. Moreover, visual help 14507 may include such video clips orpossibly an archive of problem-solving videos. In some cases, VR/MRsurgical help device 14508 may be granted control over a virtual modelor other virtual elements so that the wearer of MR/MR surgical helpdevice 14508 can present demonstrations to aid the user of MR surgicaldevice 14504. If, for example, MR surgical device 14504 presents avirtual model of an anatomical element registered to a patient's actualanatomy (e.g., a virtual model of a patient's glenoid bone registered toan actual patient's glenoid bone), VR/MR surgical help device 14508 maybe given control over a virtual model so that VR/MR surgical help device14508 can manipulate the virtual model (e.g., demonstrating a locationof a drill hole or a location of a reaming axis). In some cases, anymanipulations performed by VR/MR surgical help device 14508 on the modelmay be shown to MR surgical device 14504 as manipulations on acorresponding registered model that is registered to patient anatomy.Thus, VR/MR surgical help device may manipulate a virtual modelpresented to VR/MR surgical help device 14508 in space, and themanipulations may be presented to MR surgical device 14504 on anothervirtual model that is viewable by MR surgical device 14504 and alsoregistered to patient anatomy in the operating room and viewable by MRsurgical device 14504. In this way, for example, VR/MR surgical helpdevice 14508 could present virtual elements relative to a virtual model,and such virtual elements may be viewable by MR surgical device 14504 soas to appear relative actual patient anatomy in the operating room. Insome examples, the virtual elements presented within content 14508 mayinclude any of the virtual elements described above (or combinations ofthose described above) with regard to MR educational content 12706 ofFIG. 127.

In some cases, VR/MR surgical help device 14508 may be configured with amenu of selectable elements to assist in the interoperative education ofthe user of/MR surgical help device 14508. For example, VR/MR surgicalhelp device 14508 may be configured to present a plurality of selectablevirtual elements corresponding to a plurality of educational contentelements associated with the ongoing surgical procedure, wherein uponselection of one of the virtual elements, the second device isconfigured to educate the second user on one or more of thepreviously-executed steps associated with the surgical procedure.

Moreover, in some cases, manipulations by VR/MR surgical help device14508 on a virtual model of patient anatomy in space may appear to MRsurgical device 14504 as virtual manipulations on a registered virtualmodel that is registered to the patient anatomy in the operating room.For example, the user of VR/MR surgical help device 14508 maydemonstrate a desired location of a virtual reaming axis relative to avirtual model in space, and this demonstration may appear to MR surgicaldevice 14504 as a virtual reaming axis that is properly positionedrelative to patient anatomy since the virtual model is registered to thepatient anatomy when viewed by MR surgical device 14504. This type ofdemonstration may be very useful for the user of VR/MR surgical helpdevice 14508 to provide expert assistance to the user of MR surgicalhelp device 14504.

In some examples, a visualization device 213 may be configured toeducate a user about previously-executed steps of an orthopedic surgicalprocedure. In this example, the visualization device may correspond toVR/MR surgical help device 14508, and may comprise one or moreprocessors 514 configured to generate one or more virtual elements, anda screen 520, which may comprise a transparent mixed reality displayscreen (such as a see-through holographic lens) configured to presentthe one or more virtual elements to a user as part of a mixed realitypresentation, wherein the one or more virtual elements defineeducational content that educates the user on one or morepreviously-executed steps of the orthopedic surgical procedure. In somecases, visualization device 213 may be configured to educate the userabout the previously-executed steps of the orthopedic surgical procedurein response to a request for assistance.

As non-limiting examples, educational content that may be useful forinteroperative surgical education to educate an expert surgical helper(such as an expert surgent that is located remotely and contacted toprovide help) during a surgical procedure may include such things as:scans of a patient or segmentations image data of the patient,information associated with previously-executed surgical steps,information associated with previously-used surgical tools, informationassociated with an implanted device, information associated with a jigor guide used in the orthopedic surgical procedure, one or more imagesor videos of a prepared anatomical surface of a patient that wasprepared for an implant, information associated with a pre-operativeplan, one or more images associated with previously-executed surgicalsteps, one or more videos of previously-executed surgical steps,information associated with timing of previously-executed surgicalsteps, patient data associated with previously-executed surgical stepsanimations or illustrations of planned surgical steps, one or moreimages or videos of a surgical plan, and one or more images or videos ofthe orthopedic surgical procedure.

In some cases, a screen 520 of visualization device 213 may be sconfigured to present a plurality of selectable virtual elementscorresponding to a plurality of educational content elements, whereinupon selection of one of the virtual elements, visualization device 213is configured to inform the user on one or more of thepreviously-executed steps associated with the surgical procedure. Screen520, for example, may comprise a transparent mixed reality displayscreen, such as a see-through holographic lens.

In some examples, a method may comprise presenting a first mixed realitypresentation on a first device to aid a first user in steps of anorthopedic surgical procedure, eliciting surgical help on the orthopedicsurgical procedure, and in response to eliciting the surgical help,presenting educational content via a second mixed reality device or avirtual reality device, wherein the educational content comprisesinformation on one or more previously-executed steps of the orthopedicsurgical procedure.

From the perspective of a VR/MR surgical help device 14508, a method maycomprise receiving a request for assistance in an orthopedic surgicalprocedure, and in response to the request, presenting educationalcontent via a mixed reality device or a virtual reality device, whereinthe educational content comprises information on one or morepreviously-executed steps of the orthopedic surgical procedure. Uponreceiving and studying the educational content, the user of the VR/MRsurgical help device 14508 may be better equipped to delivering educatedexpert assistance on the orthopedic surgical procedure.

In some cases, remote users of VR/MR surgical help device 14508 may becontacted by MR surgical device 14504 or another device located in thesurgical room as part of a surgical workflow, and possibly contactedautomatically at defined steps of the surgical workflow. Local MRdevices may detect the current stage of the surgical workflow andrequest surgical assistance from a user of VR/MR surgical help device14508 for particular steps of the procedure. In this case, VR/MRsurgical help device 14508 could be contacted automatically based on thecurrent stage of the surgical workflow. In this way, high volumesurgical experts using VR/MR surgical help device 14508 can be consultedat defined stages of the surgical procedure so that they can be involvedonly when expert surgical help is desired.

In some cases, the user of VR/MR surgical help device 14508 may beconnected to MR surgical device 14504 during the surgical session at aparticular step of the procedure. Local surgical participants canmanually call the user of VR/MR surgical help device 14508 for help on aparticular step, or the call or other manner of contact to VR/MRsurgical help device 14508 could be automatic by MR surgical device14504 based on the current stage or step of the surgical workflow.

Also, if the surgical workflow process is used by MR surgical device14504 to solicit help from VR/MR surgical help device 14508, in somecases, VR/MR surgical help device 14508 may be given advance notice byMR surgical device 14504 of the need for help in the near future. Forexample, based on the current workflow of the surgical procedure, one ormore local MR devices (such as by MR surgical device 14504) may contactVR/MR surgical help device 14508 to indicate that future help is needed.As part of this request for help by MR surgical device 14504, VR/MRsurgical help device 14508 may be given advance warning or notice by MRsurgical device 14504 of when help will be needed, such as a countdownthat provides notice of when help will be needed. Artificialintelligence may be used to provide predictions of when help is needed(for defined steps or stages of a procedure) based on the current stepor stages of the surgical procedure that are currently in progress (andbased on information about previously conducted surgical procedures).Image detection and sound detection may be used by MR surgical device14504 to define or confirm the current step or stage of the surgicalprocedure so as to provide a better estimate of when the future step orstage will occur (thereby requiring, desiring, or planning for futuresurgical help by the user of VR/MR surgical help device 14508).Artificial intelligence may help guide and predict the timing forcalling the user of VR/MR surgical help device 14508 by MR surgicaldevice 14504, e.g., by MR surgical device 14504 learning the process andpredicting the timing of future steps or stages or the surgicalprocedure based on the current step or stages of the surgical procedurecurrently in progress. Recording the procedure, and identifying sounds,images, work steps, or other aspects of the procedure may allowartificial intelligence implemented by MR surgical device 14504 topredict the time when help from VR/MR surgical help device 14508 isneeded or desired. Inputs (such as sounds, images, work steps) may becorrelated with current surgical workflow steps, and this may then beused by MR surgical device 14504 to predict the timing of futureworkflow steps. This can help to provide notice or scheduling for theuser of VR/MR surgical help device 14508 and avoid down-time in thesurgical procedure when help is needed from a remote participant.

FIGS. 146 and 147 are flow diagrams illustrating inter-operativeeducational techniques that to help educate a remote user on specificdetails of an ongoing surgical procedure. As shown in FIG. 146, avisualization device 213 worn by a surgeon or another surgicalparticipant may present mixed reality to aid the user in steps of anorthopedic surgical procedure (14601). The visualization device 213 (oranother means) may elicit surgical help on the orthopedic surgicalprocedure (14602). Interoperative educational content may then beprovided to the surgical assistance via an MR device or a VR device toeducate the surgical expert on prior steps of the orthopedic surgicalprocedure (14603). In this way, the surgical expert (e.g., an expertsurgeon that is consulted intraoperatively through the use of MR or VR)can be educationally brought up to speed with regard to the procedure toensure that the surgical assistance is useful and accurate.

FIG. 147 illustrates an example method from the perspective on anexemplary VR/MR surgical help device 14508 of a surgical system 14501shown in FIG. 145. As shown in FIG. 147, VR/MR surgical help device14508 (or the user of VR/MR surgical help device 14508) receives arequest for assistance in an orthopedic surgical procedure (14701). Insome examples, VR/MR surgical help device 14508 receives the request forassistance from MR surgical device 14504. In other examples, the requestfor assistance may comprise a request to the user of VR/MR surgical helpdevice 14508, such as a telephone call, e-mail, text message, or anyother type of message from MR surgical device 14504 to the user of VR/MRsurgical help device 14508. Also, in some examples, it may be desirablefor MR surgical device 14504 to use voice-to-text technology to send atext-based request to the user of VR/MR surgical help device 14508 inresponse to a voice utterance by a user of MR surgical device 14504(e.g., a local surgeon). Voice-to-text technology may be usefulespecially when the local surgeon is unable to use his or her hands tocommunicate with the user of VR/MR surgical help device 14508. In somecases, voice-to-text technology may be used by MR surgical device 14504to deliver a written summary or the current state or the procedure, oneor more events of the procedure, surgical progress, and/or problems thathave been encountered in the surgical procedure.

In some examples, MR surgical device 14504 may be configured toautomatically send a request for help to VR/MR surgical help device14508 in response to receiving an indication of user input from the userof MR surgical device 14504. For example, the user of MR surgical device14504 may select a virtual icon or widget to cause MR surgical device14504 to send a request for help to VR/MR surgical help device 14508. Insome examples, MR surgical device 14504 may send a request for help toVR/MR surgical help device 14508 in response to detecting a voiceutterance of the user of MR surgical device 14504 that says “I want tocontact an expert” or some other trigger phrase. In examples where MRsurgical device 14504 automatically sends the request for help, some orall of the “bring up to speed” features described herein may be likewisecommunicated automatically to VR/MR surgical help device 14508.

For example, in response to the request for assistance, VR/MR surgicalhelp device 14508 is configured to present interoperative educationalcontent on prior steps of the orthopedic surgical procedure (14702). Theprior steps, for example, may include any of the operative orpre-operative steps that were already performed in relation to theorthopedic surgical procedure. Using VR/MR surgical help device 14508,an expert may then deliver educated expert assistance on the orthopedicsurgical procedure (14703).

Again, some non-limiting and non-exhaustive examples of content that maybe useful for interoperative surgical education to inform an expertsurgical helper (such as an expert surgent that is located remotely andcontacted to provide help) during a surgical procedure may include suchthings as: scans of the patient or segmentations image data of thepatient, information associated with previously-executed steps of theongoing orthopedic surgical procedure, information associated withpreviously-used surgical tools, information associated with an implanteddevice, information associated with a jig or guide used in theorthopedic surgical procedure, one or more images or videos of aprepared anatomical surface of a patient that was prepared for animplant, information associated with a pre-operative plan, one or moreimages associated with previously-executed steps of the ongoingorthopedic surgical procedure, one or more videos of previously-executedsteps of the ongoing orthopedic surgical procedure, informationassociated with timing of previously-executed steps of the ongoingorthopedic surgical procedure, patient data associated withpreviously-executed surgical steps animations or illustrations ofplanned surgical steps, one or more images or videos of a surgical plan,and one or more images or videos of the orthopedic surgical procedure.

Also, in some examples, content 14506 may comprise one or more images orvideos of the orthopedic surgical procedure, which may be captured by MRsurgical device 14504 during the orthopedic surgical procedure orcaptured by another camera or another MR device located in the operatingroom during the orthopedic surgical procedure. The images or videos maybe time-indexed. Or in some cases, the images or videos may be indexedbased on surgical steps or stages so that VR/MR surgical help device14508 an select particular surgical steps or stages and view images orvideos recorded at those steps or stages. The steps or stages manyinclude pre-operative planning steps, which are typically shown withimages, and inter-operative steps, which may be shown with video orimages. In some examples, VR/MR surgical help device 14508 may presentvirtual elements or views seen by MR surgical device 14504. In suchexamples, VR/MR surgical help device 14508 may also present a MRpre-operative planning model (e.g., above or in a side view on thepresentation seen by VR/MR surgical help device 14508).

Although VR/MR surgical help device 14508 is shown as one device, it isalso possible to have multiple VR/MR surgical help devices 14508, eachof which may be associated with different surgical experts needed atdifferent stages of the surgical procedure. Multiple VR/MR surgical helpdevices 14508 may allow for a team or community of experts to providehelp. Multiple experts may help collectively on a particular issue, oran issue may be assigned to a specific member of the team or the issuemay be assigned to a community of experts. For example, a local surgeonmay elicit help from a team of remote surgical, and a member of the teammay accept the request to provide the help via one of VR/MR surgicalhelp devices 14508. In some examples, the request for help could beassigned by MR surgical device 14504 to remote participants on afirst-come-first-serve basis. In some examples, the request for helpcould be assigned by MR surgical device 14504 to specific members of ahelp team based on the expertise of specific members of the help team.

In some examples, delivering educated expert assistance on theorthopedic surgical procedure (14703) includes presenting virtualinformation. Moreover, in some examples, the virtual information isviewable and manipulatable by a first device (VR/MR surgical help device14508) associated with a person providing the assistance and the virtualinformation is viewable relative to patient anatomy by a second device(e.g., MR surgical device 14504) associated with a person requesting theassistance. For example, VR/MR surgical help device 14508 present avirtual model in free space, which is manipulated by the user that isproviding assistance, and this virtual model may be viewable to MRsurgical device 14504 and registered by patient anatomy from theperspective of MR surgical device 14504. More generally, any virtualcontent, virtual models or virtual elements described above with regardto MR educational content 12706 of FIG. 127 could be included withincontent for orthopedic surgical help 14506, e.g., for informing the userof VR/MR surgical help device 14508 or for providing virtual-assistedhelp from the user of VR/MR surgical help device 14508 to the user of MRsurgical device 14504.

While the techniques been disclosed with respect to a limited number ofexamples, those skilled in the art, having the benefit of thisdisclosure, will appreciate numerous modifications and variations therefrom. For instance, it is contemplated that any reasonable combinationof the described examples may be performed. It is intended that theappended claims cover such modifications and variations as fall withinthe true spirit and scope of the invention.

It is to be recognized that depending on the example, certain acts orevents of any of the techniques described herein can be performed in adifferent sequence, may be added, merged, or left out altogether (e.g.,not all described acts or events are necessary for the practice of thetechniques). Moreover, in certain examples, acts or events may beperformed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors, rather than sequentially.

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof If implementedin software, the functions may be stored on or transmitted over as oneor more instructions or code on a computer-readable medium and executedby a hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media, or communication media including any mediumthat facilitates transfer of a computer program from one place toanother, e.g., according to a communication protocol. In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processors to retrieve instructions, codeand/or data structures for implementation of the techniques described inthis disclosure. A computer program product may include acomputer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. Also, any connection is properly termed acomputer-readable medium. For example, if instructions are transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, digital subscriber line (DSL), orwireless technologies such as infrared, radio, and microwave, then thecoaxial cable, fiber optic cable, twisted pair, DSL, or wirelesstechnologies such as infrared, radio, and microwave are included in thedefinition of medium. It should be understood, however, thatcomputer-readable storage media and data storage media do not includeconnections, carrier waves, signals, or other transitory media, but areinstead directed to non-transitory, tangible storage media. Disk anddisc, as used herein, includes compact disc (CD), laser disc, opticaldisc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wheredisks usually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of computer-readable media.

Operations described in this disclosure may be performed by one or moreprocessors, which may be implemented as fixed-function processingcircuits, programmable circuits, or combinations thereof, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Fixed-function circuits refer to circuits that provideparticular functionality and are preset on the operations that can beperformed. Programmable circuits refer to circuits that can programmedto perform various tasks and provide flexible functionality in theoperations that can be performed. For instance, programmable circuitsmay execute instructions specified by software or firmware that causethe programmable circuits to operate in the manner defined byinstructions of the software or firmware. Fixed-function circuits mayexecute software instructions (e.g., to receive parameters or outputparameters), but the types of operations that the fixed-functioncircuits perform are generally immutable. Accordingly, the terms“processor” and “processing circuity,” as used herein may refer to anyof the foregoing structures or any other structure suitable forimplementation of the techniques described herein.

Various examples have been described. These and other examples arewithin the scope of the following claims.

What is claimed is:
 1. A method comprising: registering, via avisualization device, a virtual model of a portion of an anatomy of anankle of a patient to a corresponding portion of the anatomy of theankle viewable via the visualization device, the virtual model obtainedfrom a virtual surgical plan for an ankle arthroplasty procedure toattach a prosthetic to the anatomy; displaying, via the visualizationdevice and overlaid on the portion of the anatomy, a virtual guide thatguides at least one of preparation of the anatomy for attachment of theprosthetic or attachment of the prosthetic to the anatomy; anddisplaying a synthesized view showing a relative position of a tibialtray trial and a tibia of the patient.
 2. The method of claim 1, whereinthe synthesized view shows a relative position of a posterior edge ofthe tibial tray and a posterior edge of the tibia.
 3. The method ofclaim 1, further comprising: registering a virtual model of the tibialtray to a portion of the tibial tray viewable via the visualizationdevice; and generating the synthesized view based on the registration ofthe virtual model of the tibial tray and the registration of the virtualmodel of the portion of the tibia.
 4. The method of claim 1, whereindisplaying the virtual guide comprises projecting the virtual guide viaa see-through lens through which a user is able to view the anatomy ofthe patient.
 5. The method of claim 1, wherein the portion of theanatomy of the patient is viewable via one or more see-through lenses ofthe visualization device, the method further comprising displaying thevirtual model and displaying the virtual guide via the see-throughlenses.
 6. The method of claim 1, wherein displaying the virtual guidecomprises displaying one or more of: a virtual axis; or a virtualcutting surface.
 7. The method of claim 1, wherein the anatomy comprisesthe tibia of the patient.
 8. The method of claim 1, wherein displayingthe virtual guide comprises: displaying a plurality of a virtual axeseach having parameters obtained from the virtual surgical plan, each ofthe virtual axes configured to guide installation of a respective guidepin in the tibia; and displaying a plurality of virtual drilling axeseach having parameters obtained from the virtual surgical plan, each ofthe virtual drilling axes configured to guide drilling of a proximalcorner of the tibia.
 9. The method of claim 1, wherein displaying thevirtual guide comprises: displaying a plurality of virtual cuttingsurfaces each having parameters obtained from the virtual surgical plan,the plurality of virtual cutting surfaces configured to guide resectionof the tibia.
 10. The method of claim 1, wherein the anatomy comprises atalus of the patient.
 11. The method of claim 1, wherein displaying thevirtual axis comprises: displaying a virtual axis having parametersobtained from the virtual surgical plan, the virtual axis configured toguide installation of a guide pin in the talus.
 12. The method of claim1, wherein displaying the virtual guide comprises: displaying aplurality of a virtual axes each having parameters obtained from thevirtual surgical plan, each of the virtual axes configured to guideinstallation of a respective guide pin in the talus.
 13. The method ofclaim 1, wherein displaying the virtual guide comprises: displaying avirtual cutting surface having parameters obtained from the virtualsurgical plan, the virtual cutting surface configured to guide resectionof the talus.
 14. A mixed reality system comprising: a memory thatstores at least a portion of a virtual surgical plan; a visualizationdevice; and one or more processors configured to: register a virtualmodel of a portion of an anatomy of an ankle of a patient to acorresponding portion of the anatomy of the ankle viewable via thevisualization device, the virtual model obtained from a virtual surgicalplan for an ankle arthroplasty procedure to attach a prosthetic to theanatomy; display, via the visualization device and overlaid on theportion of the anatomy, a virtual guide that guides at least one ofpreparation of the anatomy for attachment of the prosthetic orattachment of the prosthetic to the anatomy; and display a synthesizedview showing a relative position of a tibial tray trial and a tibia ofthe patient.
 15. The method of claim 14, wherein the synthesized viewshows a relative position of a posterior edge of the tibial tray and aposterior edge of the tibia.
 16. The method of claim 14, furthercomprising: registering a virtual model of the tibial tray to a portionof the tibial tray viewable via the visualization device; and generatingthe synthesized view based on the registration of the virtual model ofthe tibial tray and the registration of the virtual model of the portionof the tibia.
 17. The method of claim 14, wherein displaying the virtualguide comprises projecting the virtual guide via a see-through lensthrough which a user is able to view the anatomy of the patient.
 18. Themethod of claim 14, wherein the portion of the anatomy of the patient isviewable via one or more see-through lenses of the visualization device,the method further comprising displaying the virtual model anddisplaying the virtual guide via the see-through lenses.
 19. The methodof claim 14, wherein the anatomy comprises the tibia of the patient. 20.The method of claim 14, wherein displaying the virtual guide comprises:displaying a plurality of virtual cutting surfaces each havingparameters obtained from the virtual surgical plan, the plurality ofvirtual cutting surfaces configured to guide resection of the tibia. 21.The method of claim 14, wherein the anatomy comprises a talus of thepatient.
 22. The method of claim 14, wherein displaying the virtual axiscomprises: displaying a virtual axis having parameters obtained from thevirtual surgical plan, the virtual axis configured to guide installationof a guide pin in the talus.
 23. The method of claim 14, whereindisplaying the virtual guide comprises: displaying a virtual cuttingsurface having parameters obtained from the virtual surgical plan, thevirtual cutting surface configured to guide resection of the talus. 24.A computer-readable storage medium storing instructions that, whenexecuted, cause one or more processors of a mixed reality system to:register, via a visualization device, a virtual model of a portion of ananatomy of an ankle of a patient to a corresponding portion of theanatomy of the ankle viewable via the visualization device, the virtualmodel obtained from a virtual surgical plan for an ankle arthroplastyprocedure to attach a prosthetic to the anatomy; display, via thevisualization device and overlaid on the portion of the anatomy, avirtual guide that guides at least one of preparation of the anatomy forattachment of the prosthetic or attachment of the prosthetic to theanatomy; and display a synthesized view showing a relative position of atibial tray trial and a tibia of the patient.